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PREFACE 


If I could, I would reach beyond the cage of bone, 
to touch the mind within the brain, 
to reach the frightened nerves that wrap the heart; 
I would speak your name there. 


Our decisions make us who we are. Although we would like to think that our decisions 
are made rationally, deliberatively, many decisions are not. We all know that some of 
our decisions are made emotionally, and some are made reactively. Some have their 
intended consequences, and some have consequences we never imagined possible. 

We are physical beings. The human brain is a complex network of neurons and other 
cells that takes information in from the world through its sensory systems and acts on 
the world through its motor systems. But how does that network of cells, in constant 
dynamic flux, become the person you are? How does the mind fit into that small place in 
the cage of bone that is our skull? How does it process information? How does it perceive 
the world, determine the best option, select an action, and take that action? How does it 
fall in love? Laugh at the overwhelming emotion of holding an infant? How does it create 
great art or great music? How does it feel the triumphant emotion of Beethovens Ode to 
joy or the devastating pathos of Bob Dylan’s Knock Knock Knocking on Heaven’s Door ? Just 
how does the lady sing the blues? How does it get addicted and how does it break that 
addiction? How does it have a personality? What makes you you and me me? 

Fundamentally, all of these questions are about how the being that you recognize as 
yourself fits into this physical brain nestled in your skull. Fundamentally, these questions 
are about how that brain makes decisions. This book is an attempt to answer that question. 


Where this book came from 

A few years ago, John Gessner, who runs a local program for people with gambling prob¬ 
lems and their families, asked me if I would be willing to give a talk to his clients on 
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decision-making. I had been giving talks to Jan Dubinsky’s BrainU program for high 
school teachers interested in neuroscience and had recently given a talk to frontline 
medical professionals (doctors, nurses, and addiction social workers) on my labora¬ 
tory’s work identifying vulnerabilities in decision-making systems. John had heard of 
this talk and wanted me to present this work to his clients. 

I protested that I was not a medical professional and that I could not tell them how 
to fix what was broken. He said that they had lots of people to tell them that, what they 
wanted was someone to tell them why—Why do we make the choices we do ? He said they 
wanted to know why those decisions get made wrong, especially when they knew what 
the right choices were. 

The lecture itself went very well. There were several dozen people in the audience, 
and they were involved and asking questions throughout. And then, afterwards, they 
had so many questions that I stayed there answering questions for hours. There was a 
hunger there that I had not appreciated until I met those gamblers and their families, an 
almost desperate desire to understand how the brain works. They had seen how things 
can go wrong and needed an explanation, particularly one that could explain how they 
could both be conscious beings making decisions and yet still feel trapped. Somehow, 
that science lecture on how the multiple decision-making systems interact reached 
them. I realized then that there was a book I had to write. 

Over the past three years, this book has morphed and grown. My goal, however, 
remains to explain the science of how we make decisions. As such, an important part 
of this book will be to identify what questions remain . 1 My goal is not to provide a self- 
help book to help you make better decisions. I am not going to tell you what you should 
do. Nor are the answers to curing addiction or poor decisions herein. You should check 
with your own medical professionals for treatment. Every individual is unique, and 
your needs should be addressed by someone directly familiar with them. Nevertheless, 
I hope that you find the book illuminating. I hope you enjoy reading it. It has been 
tremendously fun to write. 


The structure of the book 

One of the remarkable things that has occurred over the past several decades is the con¬ 
vergence of different fields on the mechanisms of decision-making. Scientific fields as 
diverse as psychology, robotics, economics, neuroscience, and the new fields of neuro ¬ 
economics and computational psychiatry have all been converging on the recognition 
that decision-making arises from a complex interaction of multiple subsystems. In fact, 
these fields have converged on a similar categorization of the differences between the 
subsystems. In this book, we will explore how this convergence explains the decision¬ 
making that we (as humans) do. 

I have divided this book into four sections. The first sections ( Decisions and the Brain 
and The Decision-Making System ) will lay out the work that has been done on the basic 
mechanisms— What is a decision? How does the brain's decision-making system work? 
What are the components that make up that decision-making system? And then, the third 
and fourth sections will explore the consequences of that system. 
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The first section consists of five chapters, two chapters to set the stage (l: What Is 
a Decision? and 2: The Tale of the Thermostat ) and three chapters to introduce the basic 
neuroeconomics of decision-making (3: The Definition of Value, 4: Value, Euphoria, and 
the Do-It-Again Signal, and 5: Risk and Reward). In the second section, we will start 
with the results that the decision-making system is made up of multiple modules or 
subsystems (Chapter 6), and then spend a chapter each on the component systems 
(Chapters 7 through 15). 

In the third section (The Brain With a Mind of Its Own), we will explore the conse¬ 
quences of the physical nature of the brain, how mind and brain are related, and how 
vulnerabilities in the decision-making system can lead to dysfunction, such as addic¬ 
tion (Chapter 18), problem gambling (Chapter 19), and post-traumatic stress disorder 
(Chapter 20). 

Finally, in the fourth section (The Human Condition), we will explore the philosophi¬ 
cal questions of what makes us human (Chapter 22), of morality (Chapter 23), and 
of free will and consciousness (24: The Conundrum of Robotics) in the light of the new 
work on decision-making systems discussed in the previous sections. 

I’ve tried to write the book so that it can be read straight through from start to fin¬ 
ish by a reader with only a basic knowledge of neuroscience and computers; however, 
some readers may feel that they want a more detailed introduction to the concepts that 
are being discussed in this book. For those readers, I’ve included three chapters in an 
appendix, including What is information processing and How neurons process information 
(Appendix A), How we can read that information from neural signals (Appendix B), and 
How memories are stored (by content, not by index, Appendix C). 

Throughout the book, every statement is backed up with citations. These citations 
will be marked with superscript numbers, matching the list in the bibliographic notes, 
which will then reference the actual list of citations. 2 These numbers are not endnotes 
and will not be used to refer to any additional text; they are there only to back up the 
claims in the book. Instead, extra information and discussion that could distract from 
the flow will be put into footnotes, marked with superscript letters. 4 

Each chapter begins with a short poem and ends with a set of follow-up readings. 
In my mind, I think of the poems as contemplative moments that can be used to shape 
one’s perspective when reading the chapter. As a friend recovering from cancer in his 
hospital bed recently told me, “Sometimes you need the poetry to get to the heart of the 
science.” The follow-up readings at the end of each chapter are books, review articles, 
starting points for those who want to pursue a topic in depth. While the superscript cita¬ 
tions will refer to the primary literature, some of which can be quite difficult to under¬ 
stand, the follow-up readings should be understandable by anyone reading this book. 


A I prefer footnotes to endnotes because footnotes allow you to glance at the text without having 
to lose your place in the book. I will generally be including three kinds of information in footnotes: 
(1) parenthetical comments that are too long to be included in parentheses (such as the etymology and 
location of brain structures); (2) cheeky jokes and stories that I can’t resist including but would disrupt 
the flow of the book if I included them in the main text; and (3) technical details when there are subtle, 
second-order effects that need to be noted but are too complicated for those without the necessary 
background. The book should be readable without the footnotes, but I hope the footnotes will add an 
extra dimension for those who want more depth. 
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1 

What Is a Decision? 


snow flurries fall from the roof 
a squirrel skids to a stop 
Leap! into the unknown 


Inordertobeable to scientifically measure decision-making, we define 
decisions as "taking an action." There are multiple decision-making 
systems within each of us. The actions we take are a consequence of 
the interactions of those systems. Our irrationality occurs when those 
multiple systems disagree with each other. 


Because we like to think of ourselves as rational creatures, we like to define decision as the 
conscious deliberation over multiple choices. But this presumes a mechanism that might 
or might not be correct. If we are such rational creatures, why do we make such irrational 
decisions? Whether it be eating that last French fry that’s just one more than we wanted 
or saying something we shouldn’t have at the faculty meeting or the things we did that 
time we were drunk in college, we have all said and done things that we regret. Many a 
novel is based on a character having to overcome an irrational fear. Many an alcoholic has 
sworn not to drink, only to be found a few days later, in the bar, drink in hand. 

We will see later in this book that the decisions that we make arise from an interac¬ 
tion of multiple decision-making systems. We love because we have emotional reactions 
borne of intrinsic social needs and evolutionary drives (the Pavlovian system, Chapter 8). 
We decide what job to take through consideration of the pros and cons of the imagined 
possibilities (episodic future thinking and the Deliberative system, Chapter 9). We can 
ride a bike because we have trained up our Procedural learning system (Chapter 10), but 
it can also be hard to break bad habits that we’ve learned too well (Chapter 15). We will 
see that these are only a few of the separable systems that we can identify. All of these 
different decision-making systems make up the person you are. 

The idea that our actions arise from multiple, separably identifiable components 
has a long history in psychology, going back to Freud, or earlier, and has gained recent 
traction with theories of distributed computing, evolutionary psychology, and behav¬ 
ioral economics. 1 Obviously, in the end, there is a single being that takes an action, but 
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sometimes it’s helpful to understand that being in terms of its subsystems. The anal¬ 
ogy that I like, which I will use several times in this book, is that of a car. The car has a 
drive train, a steering system, brakes. Cars often have multiple systems to accomplish 
the same goal (the foot-pedal brake and the emergency/parking brake, or the electric 
and gasoline engines in a hybrid like the Toyota Prius). 

The psychologist Jonathan Haidt likes to talk of a rider and an elephant as a meta¬ 
phor for the conscious and unconscious minds both making decisions, 2 but I find this 
analogy unsuitable because it separates the “self” from the “other” decision-making 
systems. As is recognized by Haidt at the end of his book, you are both the rider and 
the elephant. When a football star talks about being “in the zone,” he’s not talking about 
being out of his body and letting some other being take over—he feels that he is making 
the right decisions, (in fact, he’s noticing that his procedural/habit system is working 
perfectly and is making the right decisions quickly and easily.) When you are walking 
through a dark forest and you start to get afraid and you jump at the slightest sound, 
that’s not some animal reacting—that’s you. (it’s a classic example of the Pavlovian 
action-selection system.) Sometimes these systems work together. Sometimes they 
work at cross purposes. In either case, they are all still you. “Do I contradict myself?” 
asks Walt Whitman in his long poem Song of Myself. “Then I contradict myself. I am 
large. I contain multitudes.” 

So how do we determine how these multiple systems work? How do we determine 
when they are working together and when they are at cross-purposes? How do we iden¬ 
tify the mechanisms of Whitman’s multitudes? 

To study something scientifically, we need to define our question in a way that we 
can measure and quantify it. Thus, we need a measure of decision-making that we can 
observe, so we can compare the predictions that arise from our hypotheses with actual 
data. This is the key to the scientific process: there must be a comparison to reality. If the 
hypothesis doesn’t fit that reality, we must reject the hypothesis, no matter how much 
we like it. 

One option is to simply to ask people what they want. But, of course, the idea that 
we always do what we say we want makes very strong assumptions about how we make 
decisions, and anyone who has regretted a decision knows that we don’t always decide 
to do what we want. Some readers may take issue with this statement, saying that you 
wanted that decision when you took the action. Just because you regret that night of 
binge drinking when you have a hangover in the morning doesn’t mean you didn’t want 
all those drinks the night before. Other readers may argue that a part of you wanted 
that decision, even if another part didn’t. We will come to a conclusion very similar 
to this, that there are multiple decision-making modules, and that the members of 
Whitman’s multitudes do not always agree with each other. Much of this book will be 
about determining who those multitudes are and what happens when they disagree 
with each other. 

As a first step in this identification of the multitudes, careful scientific studies have 
revealed that a lot of conscious “decisions” that we think we make are actually rational¬ 
izations after the fact. 3 For example, the time at which we think we decided to start an 
action is often after the action has already begun. 

In what are now considered classic studies of consciousness, Benjamin Libet asked 
people to perform an action (such as tapping a finger whenever they wanted to) while 
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watching a dot move around a circle. 4 The people were asked to report the position 
of the dot when they decided to act. Meanwhile, Libet and his colleagues recorded 
electrical signals from the brain and the muscles. Libet found that these signals pre¬ 
ceded the conscious decision to act by several hundred milliseconds. Both the brain 
and muscles work by manipulating electricity which we can measure with appro¬ 
priate technologies. With the appropriate mathematics, we can decode those signals 
and determine what is happening within the brain. Libet decoded when the action 
could be determined from signals in the motor areas of the brain and compared it to 
when consciousness thought the action had occurred. Libet found that the conscious 
decision to take an action was delayed, almost as if consciousness was perceiving the 
action, rather than instigating it. Several researchers have suggested that much of con¬ 
sciousness is a monitoring process, allowing it to keep track of things and step in if 
there are problems. 5 

Much of the brains machinery is doing this sort of filling-in, of making good guesses 
about the world. Our eyes can focus on only a small area of our visual field at a time. 
Our best visual sensors are an area of our retina that has a concentration of cells tuned 
to color and specificity in a location called the fovea. This concentration of detectors at 
the fovea means that were much better at seeing something if we focus our eyes on it. 
Our vision focuses on a new area of the visual world every third of a second or so. The 
journeys between visual focusings are called saccades. Even while we think we are focus¬ 
ing our attention on a small location, our eyes are making very small shifts called micro- 
saccades. If the visual world is aligned to our microsaccades so that it shifts when we do, A 
the cells adapt to the constant picture and the visual image “grays out” and vanishes. 
This means that most of our understanding of the visual world is made from combining 
and interpreting short-term visual memories. We are inferring the shape of the world, 
not observing it. 

In a simple case familiar to many people, there’s a small location on our retina where 
the axons from the output cells have to pass through to form the optic nerve sending the 
visual signals to the rest of our brain. This leaves us with a “blind spot” that must be filled 
in by the retina and visual processing system.® Our brain normally fills in the “blind 
spot” from memories and expectations from the surrounding patterns. 7 

In a similar way, we don’t always notice what actually drives our decision-making 
process. We rationalize it, filling in our reasons from logical perspectives. Some of my 
favorite examples of this come from the work ofV. S. Ramachandran, 8 who has studied 
patients with damage to parts of the brain that represent the body. A patient who is 


A The visual world can be aligned to our microsaccades by tracking the movement of the eyes and 
shifting a video display very quickly. This is feasible with modern computer technology. 

B The wires (axons) of our optic nerve have to pass through the retina because the retina is built 
backwards, with the processing cells on top and the light-detecting cells on the bottom. The process¬ 
ing cells are generally transparent, so light passes through them and the light-detecting cells can still 
see, but at some point the axons have to leave the processing cells to travel to the brain. There are no 
light-detecting cells at the point where the axons pass through, leaving a blind spot. This is a relic of 
the evolutionary past of the human species. Eyes do not have to be built this way—the octopus eye, 
with a different evolutionary history, is oriented correctly, with the light-detecting cells on top and the 
processing cells below. Octopi therefore do not have a blind spot that needs to be filled in. 6 
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physically unable to lift her arm denies that she has a problem and merely states that she 
does not want to. When her arm was made to rise by stimulating her muscles directly 
she claimed that she had changed her mind and raised her arm because she wanted to, 
even though she had no direct control of the arm. In a wonderful story (told by Oliver 
Sacks in A Leg to Stand On), a patient claims that his right hand is not his own. When 
confronted with the fact that there are four hands on the table (two of his and the two 
of the doctor’s), the patient says that three of the hands belong to the doctor. “How 
can I have three hands?” asks the doctor. “Why not? You have three arms.” replies the 
patient. 

In his book Surely You’re Joking, Mr. Feynman, the famous physicist Richard Feynman 
described being hypnotized. He wrote about how he was sure he could take the action 
(in this case opening his eyes) even though he had been hypnotized not to, but he 
decided not to in order to see what would happen. So he didn’t, which was what he 
had been asked to do under hypnosis. Notice that he has rationalized his decision. As 
Feynman himself recognized, even if he said “I could have opened my eyes,” he didn’t. 
So what made the decision? Was it some effect of the hypnosis on his brain or was it that 
he didn’t want to? How could we tell? Can we tell? 

Many of the experiments we’re going to talk about in this book are drawn from 
animals making decisions. If we’re going to say that animals make decisions, we need 
to have a way of operationalizing that decision—it’s hard to ask animals what they 
think. There are methods that allow us to decode information represented within 
specific parts of the brain, which could be interpreted as a means of asking an ani¬ 
mal what it thinks (see Appendix B). However, unlike asking humans linguistically, 
where one is asking the overall being what it thinks, decoding is asking a specific 
brain structure what it is representing. Of course, one could argue that assuming 
what people say is what they think assumes that humans are unified beings. As we 
will see as we delve deeper into how decisions are made, humans (like other mam¬ 
mals) are mixtures of many decision-making systems, not all of which always agree 
with each other. 

Just as the Libet experiments suggest that parts of the brain can act without con¬ 
sciousness, there are representations in the brain that are unable to affect behavior. 
In a remarkable experiment, Pearl Chiu, Terry Lohrenz, and Read Montague found 
signals in both smokers’ and nonsmokers’ brains that represented not only the suc¬ 
cess of decisions made but also what they could have done if they had made a bet¬ 
ter choice. 9 This recognition of what they could have done is called a counterfactual 
(an imagination of what might have been) and enables enhanced learning, (it is now 
known that both rats and monkeys can represent counterfactual reward information 
as well. These signals appear to use the same brain structures that Chiu, Lohrenz, 
and Montague were studying, and to be the same structures involved when humans 
express regret. 10 ) Counterfactuals enhance learning by allowing one to learn from 
imagined possibilities. For example, by watching someone else make a mistake. Or 
(in the example used by Chiu, Lohrenz, and Montague) “if I had taken my money out 
of the stock market last week, I wouldn’t have lost all that money when it crashed.” 
While signals in both groups’ brains reflected this counterfactual information, only 
nonsmokers’ behavior took that information into account. If we want to under¬ 
stand how decisions are made and how they go wrong, we are going to need a way to 
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determine not only the actions taken by the subject but also the information process¬ 
ing happening in his or her brain. 

Certainly most nonhuman animals don’t have language, although there may be some 
exceptions. c Nevertheless, it would be hard to ask rats, pigeons, or monkeys (all of which 
we will see making decisions later in the book) what they want linguistically. Given that 
it is also hard to ask humans what they really want, we will avoid this language problem 
altogether and operationalize making a decision as taking an action, because taking an 
action is an observable response. 

This is very similar to what is known in behavioral economics as revealed preferences. 13 
Economic theory (and the concomitant new field of neuroeconomics) generally 
assumes that those revealed preferences maintain a rational ordering such that if you 
prefer one thing (say chocolate ice cream) to another (say vanilla ice cream), then you 
will always prefer chocolate ice cream to vanilla if you are ever in the same situation. 
We will not make that assumption. 

Similarly, we do not want to assume that a person telling you what he or she wants 
actually reflects the choices a person will make when faced with the actual decision. 14 
Given the data that our conscious observations of the world are inferred and the data 
that our spoken explanations are rationalizations, 15 some researchers have suggested 
that our linguistic explanations of our desires are better thought of as the speech of a 
“press secretary” than the actions of an executive. 16 Thus, rather than asking what some¬ 
one wants, we should measure decisions by giving people explicit choices and asking 
them to actually choose. We will encounter some of the strangenesses discovered by 
these experiments in subsequent chapters. 

Often, experimentalists will offer people hypothetical choices. It is particularly dif¬ 
ficult to get funding to provide people a real choice between $10,000 and $100,000, 
or to give them a real choice whether or not to kill one person to save five. Instead, 
subjects are asked to imagine a situation and pretend it was real. In practice, in the few 
cases where they have been directly compared, hypothetical and real decision-making 
choices tend to match closely. 17 But there are some examples where hypothetical and 
real decisions diverge. 18 These tend to be with rewards or punishments that are sensory, 
immediate, and what we will recognize later as Pavlovian (Chapter 8). 

A classic example of this I call the parable of the jellybeans. Sarah Boysen and Gary 
Berntson tried to train chimpanzees to choose the humble portion. 19 They offered the 
subject two trays of jellybeans. If he reached for the larger tray, he got the smaller one 
and the larger one was given to another chimpanzee; however, if the deciding chimpan¬ 
zee reached for the smaller tray, he got the larger tray and the smaller one was given to 
another chimpanzee. When presented with symbols (Arabic numerals that they had 
previously been trained to associate with numbers of jellybeans), subjects were able 


c The extent to which animals can acquire human-level languages is not without its controversies. 
Since were not going to trust human language either, 11 this issue is actually irrelevant to our question of 
decision-making. For those interested in the question of animals learning language, I recommend one of 
the excellent books written by the various researchers who have tried to teach language to nonhuman ani¬ 
mals, such as Daniel Fouts teaching Washoe the chimpanzee, Penny Patterson teaching Koko the gorilla, 
Sue Savage-Rumbaugh teaching Kanzi the bonobo, or Irene Pepperberg teaching Alex the parrot. 12 
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to choose the smaller group, but when the choices were physical jellybeans, they were 
unable to prevent themselves from reaching for the larger group of jellybeans. Other 
experiments have found that more linguistically capable animals are more able to per¬ 
form these self-control behaviors. 20 This may be akin to our ability to talk ourselves out 
of doing things that we feel we really want: “I know I’m craving that cigarette. But I don’t 
want it. I really don’t.” 

A similar experiment is known colloquially as the marshmallow experiment. 11 Put a 
single marshmallow in front of a child sitting down at the kitchen table. Tell the child 
that if the marshmallow is still sitting there in five minutes, you’ll add a second marsh¬ 
mallow to it. Then leave the room. It is very, very difficult for children not to reach for 
the marshmallow. It is much easier for children to wait for two pennies or two tokens 
than for two marshmallows. We will discuss the marshmallow experiment in detail in 
the chapter on self-control (Chapter 15). 

Studies of decision-making in psychology (such as the marshmallow experiment) 
as well as studies in behavioral economics and the new field of neuroeconomics tend to 
measure choices within the limitation of discrete options. In the real world, we are rarely 
faced with a discrete set of options. Whether it be deciding when to swing the bat to 
hit a baseball or deciding where to run to on a playground, we are always continuously 
interacting with the world. 22 As we will see later, some of the mechanisms that select the 
actions we take are not always deliberative, and do not always entail discrete choices. 

So where does that leave us in our search for a definition of decision-making? 
We will not assume that all decision-making is rational. We will not assume that all 
decision-making is deliberative. We will not assume that decision-making requires lan¬ 
guage. Instead, we define decision-making as the selection of an action. Obviously, many 
of the decisions we take (such as pushing a lever or button—say on a soda machine) 
are actual actions. But note that even complex decisions always end in taking an action. 
For example, buying a house entails signing a form. Getting married entails making a 
physical statement (saying “I do”). We are going to be less concerned about the physi¬ 
cal muscle movements of the action than about the selection process that decided on 
which action to take. Nevertheless, defining “decision-making” as “action-selection” will 
force us to directly observe the decisions made. It will allow us to ask why we take the 
actual actions we do. Why don’t those actions always match our stated intentions? How 
do we choose those actions over other potential actions? What are the systems that 
select actions, and how do those systems interact with the world? How do they break 
down? What are their vulnerabilities and failure-modes? That is what the rest of this 
book is about. 
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jet exhaust shimmers above the tarmac 

I remember the strength of swans 
on the lake up North, their wings 
stretching forward, beating back... 

acceleration pushes us into our seats 
and we lift into the sky 


Your brain is a decision-making machine, a complex but physical 
thing. Like any physical process, there are multiple ways in which 
decisions can go wrong. Being a physical being does not diminish who 
you are, but it can explain some of the irrational choices you make. 


The decision-making that you do arises from the physical processes that occur in your 
brain. Because your brain is a physical thing, it has vulnerabilities, what one might call 
“failure-modes” in the engineering world. We see these vulnerabilities in our susceptibil¬ 
ity to bad choices (Do you really need a new sports car?), in our susceptibility to addictions 
( Why can’t you just quit smoking those cigarettes ?), in our inability to control our emotions 
or our habits (Why do you get so angry about things you can’t change?). We would like to 
believe that we are rational creatures, capable of logic, always choosing what’s best for 
us. But anyone who has observed their own decisions (or those of their friends) will rec¬ 
ognize that this is simply not correct. We are very complex decision-making machines, 
and sometimes those decision-making processes perplex us. Understanding how the 
brain makes decisions will help us understand ourselves. To understand those vulner¬ 
abilities, we need to understand the mechanism of decision-making in our brains. 

Today, we are all familiar with complex machines, even complex machines that 
make decisions. The simplest machine that makes a decision is the thermostat—when 
the house is too hot, the thermostat turns on the air conditioning to cool it down, and 
when the house is too cold, the thermostat turns on the furnace to heat it up. This 
process is called negative feedback —the thermostat’s actions are inversely related to the 
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difference between the temperature of the room and the temperature you’d like it to be 
(the set-point). But is the process really so simple? Taking the decision-making process 
of the thermostat apart suggests that even the simple decision-making process of the 
thermostat is not so simple. 

The thermostat has three key components of decision-making that we will come 
back to again and again in this book. First, it perceives the world —the thermostat has 
a sensor that detects the temperature. Second, it determines what needs to be done —it 
compares that sensor to the set-point and needs to increase the temperature because it 
is too cold, needs to decrease the temperature because it is too hot, or doesn’t need to 
do anything because the temperature is just right. Finally, it takes an action —it turns on 
either the furnace or the air conditioning. 

In the artificial intelligence literature, there was an argument through much of the 
1980s about whether a thermostat could have a “belief.” 1 Fundamentally, a belief is a 
(potentially incorrect) representation about the world. Clearly, a working thermostat 
requires a representation of the target temperature in order to take actions reflecting 
the temperature of the outside world. But can we really say that the thermostat “rec¬ 
ognizes” the temperature of the outside world? The key to answering this question is 
that the thermostat does not take actions based on the temperature of the world, but 
rather on its internal representation of the temperature of the world. Notice that the 
internal representation might differ from the real temperature of the room. If the sensor 
is wrong, the thermostat could believe that the room is warmer or cooler than it really is 
and take the wrong action. 

One of the key points in this book is that knowing how the brain works allows us a 
better perception of what happens when something goes wrong. I live in Minnesota. In 
the middle of winter, it can get very cold outside. Imagine you wake up one morning to 
find your bedroom is cold. Something is wrong. But simply saying that the thermostat 
is broken won’t help you fix the problem. We need to identify the problem, to diagnose 
it, if you will. 

Maybe something is wrong with the thermostat’s perception. Perhaps the sensoris bro¬ 
ken and is perceiving the wrong temperature. In this case, the thermostat could think that 
the house is fine even though it is too cold. (Notice the importance of belief here—the dif¬ 
ference between the thermostat’s internal representation and the actual temperature can 
have a big impact on how well the thermostat makes its decision!) Maybe the set-point is 
set to the wrong temperature. This means that the thermostat is working properly—it has 
correctly moved the temperature of your house to the set-point, but that’s not what you 
wanted. Or, maybe there’s something wrong with the actions available to the thermostat. 
If the furnace is broken, the thermostat may be sending the signal saying “heat the house” 
but the house would not be heating correctly. Each of these problems requires a differ¬ 
ent solution. Just as there are many potential reasons why your bedroom is too cold and 
knowing how a thermostat works is critical to understanding how to fix it, when smokers 
say that they really want to quit smoking, but can’t, we need to know where each indi¬ 
vidual’s decision-making process has gone wrong or we won’t be able to help. Before we 
can identify where the decision-making process has broken down, we’re going to need to 
understand how the different parts of the brain work together to make decisions. 

Many readers will object at this point that people are much more complicated 
than thermostats. (And we are.) Many readers will then conclude that people are not 
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machines. Back when negative feedback like the thermostat was the most complicated 
machine mechanism that made decisions, it was easy to dismiss negative feedback as too 
simple a model for understanding people. However, as we will see later in the book, we 
now know of much more complicated mechanisms that can make decisions. Are these 
more complicated mechanisms capable of explaining human decision-making? (I will 
argue that they are.) This leaves open some difficult questions: Can we be machines and 
still be conscious? Can we be machines and still be human? 

The concept of conscious machines making decisions pervades modern science fic¬ 
tion, including the droids C3P0 and R2D2 in Star Wars, the android Data of Star Trek: 
The Next Generation, the desperate Replicants of Ridley Scott’s Blade Runner, and the 
emotionally troubled Cylons ofBattlestar Galactica. Star Trek: The Next Generation spent 
an entire episode (The Measure of a Man ) on the question of how the fact that Data was 
a machine affected his ability to decide for himself whether or not to allow himself to be 
disassembled. In the episode, the judge concludes the trial with a speech that directly 
addresses this question—“Is Data a machine? Yes. Is he the property of Starfleet? No. 
We have all been dancing around the basic issue: does Data have a soul? I don’t know 
that he has. I don’t know that I have. But I have got to give him the freedom to explore 
that question himself. It is the ruling of this court that Lieutenant Commander Data has 
the freedom to choose.” 2 We will examine the complex questions of self and conscious¬ 
ness in detail at the end of the book (The Conundrum of Robotics, Chapter 24), after we 
have discussed the mechanisms of decision-making. In the interim, I aim to convince 
you that we can understand the mechanisms of our decision-making process without 
losing the freedom to choose. 

I don’t want you to take the actual process of the thermostat as the key to the story 
here, anymore than we would take jet planes as good models of how swans fly. And 
yet, both planes and swans fly through physical forces generated by the flow of air over 
their specially shaped wings. Even though bird wings are bone, muscle, and feathers, 
while airplane wings are metal, for both birds and planes, lift is generated by airflow 
over the wings, and airflow is generated by speed through the air. If we can understand 
what enables a 30-ton airplane to fly, we will have a better understanding of how a 
30-pound swan can fly. Even though the forward push through the air is generated dif¬ 
ferently, both birds and planes fly through physical interactions with the air. We will 
use analogous methods of understanding decision-making processes to identify and fix 
problems in thermostats and in ourselves, because both use identifiable computational 
decision-making processes. 

A good way to identify where a system (like a thermostat) has broken down is a pro¬ 
cess called “differential diagnosis,” as in What are the questions that will differentiate the 
possible diagnoses? My favorite example of this is the show CarTalk on National Public 
Radio, in which a pair of MIT-trained auto mechanics (Tom and Ray Magliozzi) diag¬ 
nose car troubles. When a caller calls in with a problem, the first things they discuss are 
the basics of the problem. (Anyone who has actually listened to CarTalk will know that 
the first things Tom and Ray discuss are the person’s name, where the caller is from, 
and some completely unrelated jokes. But once they get down to discussing cars, they 
follow a very clear process of differential diagnosis.) A typical call might start with the 
caller providing a description of the problem—“I hear a nasty sound when I’m driv¬ 
ing.” And then Tom and Ray will get down to business—they’ll ask questions about 
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the sound: “What is the sound? Where is it coming from? Does it get faster as you go 
faster? Does it still happen when the engine is on but the car is not moving?” Each ques¬ 
tion limits the set of possible problems. By asking the right series of questions, one can 
progressively work one’s way to identifying what’s wrong with the car. If we could orga¬ 
nize these questions into a series of rules, then we could write a computer program to 
solve our car problems. (Of course, then we wouldn’t get to hear all the CarTalk jokes. 
Whether this is good or bad is a question of taste.) 

In the 1980s, the field of artificial intelligence developed “expert systems” that codi¬ 
fied how to arrange these sorts of question-and-answer rules to perform differential 
diagnosis. 3 At the time, expert systems were hailed as the answer to intelligence—they 
could make decisions as well as (or better than) experts. But it turns out that most 
humans don’t make decisions using differential diagnoses. In a lot of fields (includ¬ 
ing, for example, medicine), a large part of the training entails trying to teach people 
to make decisions by these highly rational rules. 4 However, just because it is hard for 
people to make decisions by rule-based differential diagnoses does not mean that 
humans don’t have a mechanism for making decisions. In fact, critics complained that 
the expert systems developed by artificial intelligence were not getting at the real ques¬ 
tion of what it means to be “intelligent” long before it was known that humans didn’t 
work this way. 5 People felt that we understood how expert systems work and thus 
they could not be intelligent. A classmate in college once said to me that “we would 
never develop an artificial intelligence. Instead, we will recognize that humans are not 
intelligent.” One goal of this book is to argue that we can recognize the mechanisms 
of human decision-making without losing our sense of wonder at the marvel that is 
human intelligence. 

Some readers will complain that people are not machines; they have goals, they have 
plans, they have personalities. Because we are social creatures and much of our intelli¬ 
gence is dedicated to understanding each other, we have a tendency to attribute agency 
to any object that behaves in a complex manner. 6 Many of my friends name their cars 
and talk about the personality of their cars. My son named our new GPS navigation 
system “Dot.” When asked why he named the GPS (the voice is definitely a woman’s), 
he said, “So we can complain to her when she gets lost—‘Darn you, Dot!”’ 

A GPS navigator has goals. (These are goals we’ve programmed in, but they are goals 
nonetheless.) Dot’s internal computer uses her knowledge of maps and her calcula¬ 
tion of the current location to make plans to achieve those goals. You can even tell Dot 
whether you prefer the plans to include more highways or more back-country scenic 
roads. If Dot were prewired to prefer highways or back-country scenic roads, we would 
say she has a clear personality. In fact, I wish I had some of Dot’s personality—when we 
miss an exit, Dot doesn’t complain or curse, she just says “recalculating” and plans a new 
route to her goal. 

The claim that computers can have goals and that differences in how they reach those 
goals reflects their personality suggests that goals and plans are simple to construct and 
that personality is simply a difference in underlying behavioral variables and prefer¬ 
ences. We explain complex machines by thinking they’re like people. Is it fair to turn that 
on its head and explain people as complex machines? 
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In this book, I’m going to argue that the answer to this question is “yes ”—your brain 
is a decision-making machine, albeit a very complex one. You are that decision-making 
machine. This doesn’t mean you’re not conscious. This doesn’t mean you’re not you. But 
it can explain some of the irrational things that you do. 

Understanding how the human decision-making system works has enormous impli¬ 
cations for understanding who we are, what we do, and why we do what we do. Scientists 
study brains, they study decision-making, and they study machines. By bringing these 
three things together, we will begin to get a sense of ourselves. In this book, I will discuss 
what we know about how brains work, what we know about how we make decisions, 
and what we know about how that decision-making machine can break down under 
certain conditions to explain irrationality, impulsivity, and even addiction. 
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The Definition of Value 


diamond stars in a circle of gold 
declare your love 
with two months salary 


Value measures how much one is willing to pay, trade, or work for 
a reward, or to work to avoid a punishment. Value is not intrinsic 
to an object, but must be calculated anew each time. This produces 
inconsistencies, in which different ways of measuring value can 
produce different orderings of one’s preferences. 


What is “value”? What does it mean to value something? In psychological theories, value 
is thought of as that which reduces needs or which alleviates negative prospects 1 —when 
you are hungry food has value; when you are thirsty water has value. Pain signals a neg¬ 
ative situation that must be alleviated for survival. In economics, the concept of revealed 
preferences says that we value the things we choose, and make decisions to maximize 
value. 2 Although this is a circular definition, we will use it later to help us understand 
how the decision-making process operates. In the robotics and computer simulation 
worlds where many of the mechanisms that we will look at later have been worked out, 
value is simply a number r for reward or -r for punishment. 3 The robotics and computer 
science models simply try to maximize r (or minimize -r). 

In many experiments that study decision-making, value is measured in terms of money. 
But of course, money has true value only in terms of what it can buy. 4 Economists will often 
tell us that money is an agreed-upon fiction—I am willing to sell you this apple for a green 
piece of paper that we both agree is worth $ 1 because I am confident that I can then use the 
paper later to buy something that I want. The statement that money is valuable only in terms 
of what it can buy is not entirely true either: money can also have value as a symbol of what 
it implies for our place in the social network. 5 This is one reason why extremely well-paid 
professional athletes fight for an extra dollar on their multimillion dollar salaries—they 
want to be known as the “best-paid wide receiver” because that identification carries social 
value. Money and value are complex concepts that interact in difficult ways. 
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Part of the problem is that you cannot just ask people How much do you value this 
thing? We don’t have units to measure it in. Instead, value is usually measured in what 
people will trade for something and in the decisions they make. 6 This is the concept 
called revealed preferences —by examining your decisions, we can decide what you value. 
Theoretically, this is very simple: one asks How much will you pay for this coffee mug? The 
price you’ll pay is the value of the coffee mug. If we measure things at the same time, in 
the same experiment, we shouldn’t have to worry about inflation or changes in the value 
of the money itself. 

The problem is that people are inconsistent in their decisions. In a famous experi¬ 
ment, Daniel Kahneman and his colleagues Jack Rnetsch and Richard Thaler divided 
their subjects into two groups: one group was asked how much money they would pay 
for a $6 (in 1990) Cornell University coffee mug and the other group was first given the 
coffee mug and then asked how much money they would accept for the mug. The first 
group was willing to pay much less on average ($3) than the second group was willing 
to accept for it ($7). This is called the endowment effect and is a preference for things you 
already own. 7 

Simply phrasing decisions in terms of wins or losses changes what people choose. 8 
The interesting thing about this is that even once one is shown the irrationality, it still 
feels right. This effect was first found by Daniel Kahneman and Amos Tversky, who 
made this discovery sitting in a coffee shop in Israel, asking themselves what they would 
do in certain situations. When they found themselves making irrational decisions, they 
took their questions to students in their college classes and measured, quantitatively, 
what proportion made each decision in each condition. 

In the classic case from their 1981 paper published in the journal Science: 9 

Imagine that the U.S. is preparing for the outbreak of an unusual Asian disease, 
which is expected to kill 600 people. Two alternative programs to combat the 
disease have been proposed. Assume that the exact scientific estimate of the 
consequences of the programs are as follows: 

Problem 1: If Program A is adopted, 200 people will be saved. If Program B 
is adopted, there is 1/3 probability that 600 people will be saved, and 2/3 
probability that no people will be saved. Which of the two programs would 
you favor? 

Problem 2: If Program C is adopted, 400 people will die. If Program D is 
adopted, there is 1/3 probability that nobody will die, and 2/3 probability that 
600 people will die. Which of the two programs would you favor? 

A careful reading of the two conditions shows that Program C is identical to 
Program A, while Program D is identical to Program B. Yet, 72% of the first group chose 
program A and 28% chose program B, while only 22% of the second group chose pro¬ 
gram C and 78% chose program D! Kahneman and Tversky interpret this as implying 
that we are more sensitive to losses than to gains—we would rather risk not gaining 
something than we would risk losing something. This is clearly a part of the story, but I 
will argue that there is a deeper issue here. I will argue that “value” is something that we 
calculate each time, and that the calculation process doesn’t always come up with the 
same “rational” answer. 
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Illogical value calculations 

The most interesting thing about these results (of which there are many) is that even 
when they are illogical (like the disease example above), they often still sound right. 
There are other examples where we can see that decisions are irrational, and don’t sound 
right when they are pointed out to us, but which humans definitely show when tested. 
The best examples of most of these come from the advertising and marketing worlds. 

A person goes into an electronics store to buy a television and finds three televisions 
on sale, one for $100, one for $200, and one for $300. Imagine that the three televisions 
have different features, such that these are reasonable prices for these three TVs. People 
are much more likely to buy the $200 TV than either of the other two. If, in contrast, 
the person goes into the store and finds three televisions on sale, the same $200 TV, the 
same $300 TV, and now a fancier $400 TV, the person is more likely to buy the $300 
TV. In the first case, the person is saying that the $200 TV is a better deal than the $300 
one, but in the second case, the person is saying that the $300 TV is the better deal. 
Even though they’ve been offered the same televisions for the same prices, the decision 
changed depending on whether there is a $100 TV or a $400 TV in the mix. This is 
called extremeness aversion and is a component of a more general process called framing. 
In short, the set of available options changes your valuation of the options. This is com¬ 
pletely irrational and (unlike the other examples above) seems unreasonable (at least to 
me). Yet it is one of the most reliable results in marketing and has probably been used 
since the first markets in ancient times through to the modern digital age. 10 

Products in advertisements used to compare themselves to each other. Tylenol would 
say it was better than aspirin and Coke would say it was better than Pepsi. I remember 
an RC Cola commercial with two opposing teams fighting about which drink they pre¬ 
ferred, Coke or Pepsi, while a third person sits on the sidelines drinking an RC Cola, 
out of the fray, smiling. Advertisements today rarely mention the competition. This is 
because one of the heuristics we use is simply whether we recognize the name or not. 11 
So although RC Cola was trying to communicate that Coke and Pepsi were the same, 
while RC Cola was different, what people took from the advertisement was a reminder 
that everyone drank Coke and Pepsi. Just mentioning the name reminds people that it 
exists and reinforces the decision to choose it. Familiarity with a brand name increases 
the likelihood that one will select it. 

It is election season as I write this. All around my neighborhood, people have put 
out signs for their candidates. The signs don’t say anything about the candidates. Often 
they don’t even have the party the candidates belong to. Usually, it’s just the name of the 
candidate, and sometimes, an appeal to “vote for” the candidate. What information do 
I get from seeing a sign that says nothing other than “Vote for X”? I suppose that one 
may need to be reminded to vote at all, but then why does the sign include “for X” on 
it? (One of my neighbors does have a large handmade sign she puts out every election 
season that just says “VOTE!” on it.) It is true that one can get a sense of the group¬ 
ing of candidates from people who put out multiple signs: given that I like person X 
for state representative, seeing that all the people with person X also have person Y for 
county prosecutor, while all the people who have person X’s opponent have person Z 
for county prosecutor, might suggest to me that I would like person Y over person Z for 
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county prosecutor. But lots of people have just one sign out. All that tells me is that lots 
of people like person X. Why would knowing that lots of people like something suggest 
that I would too? 

There are actually three things that these single-sign houses are doing. First, they are 
increasing my familiarity with that name. As with the products, just knowing the name 
increases one’s likelihood of voting for someone. Second, if lots of people like a movie, 
it’s more likely to be a good movie than one everybody hated. Using the same heuristic, 
if everyone likes a candidate, isn’t that candidate more likely to be a good choice? And 
third, we like to be on the winning team. If everyone else is going to vote for someone, 
he or she is likely to win. Of course, we’re supposed to be voting based on who we think 
is a better choice to govern, not who is most popular. But it’s pretty clear that a lot of 
people don’t vote that way. 

These effects occur because we don’t actually calculate the true value of things. Instead, 
we use rules of thumb, called heuristics, that allow us to make pretty good guesses at how 
we value things. 12 If you like all the choices, picking the middle one is a pretty good guess 
at good value for your money. If you’re making a decision, familiarity is a pretty good 
guess. ( Something you’re familiar with is likely to be something you’ve seen before. If you 
remember it, but don’t remember it being bad, how bad could it have been?) 

Some economists have argued that evolutionarily, heuristics are better than actually 
trying to calculate the true value of things because calculating value takes time and heu¬ 
ristics are good enough to get us by. 13 But a lot of these nonoptimal decisions that we’re 
making are taking time. Knowing that Programs A and C are the same and that programs 
B and D are the same in the Kahneman and Tversky flu example above doesn’t change 
our minds. This makes me suspect that something else is going on. It’s not that heuristics 
are faster and we are making do with “good enough.” I suspect that these effects have to 
do with how we calculate value. We cannot determine how humans calculate value unless 
we can measure it. So, again, we come back to the question of how we measure value. 


Measuring value 

With animals, we can’t ask them how much they value something; we can only offer 
them something and determine how much they’ll pay for it. Usually, this is measured in 
terms of the amount of effort an animal will expend to get the reward. 14 How many lever 
presses is the animal willing to make for each reward? 

Alternatively, we can give the animal a direct choice between two options: 15 Would 
the animal rather have two apple-flavored food pellets or one orange-flavored food pellet? 
Would it rather have juice or water? We can also titrate how much of one choice needs to 
be offered to make the animal switch—if the animal likes apple flavor better than orange 
flavor, would it still prefer half as much apple to the same amount of orange? 

Finally, we can also measure the negative consequences an animal will put up with 
to get to a reward. 16 Usually, this is measured by putting a negative effect (a small shock 
or a hot plate) in between the animal and the reward, which it has to cross to get to 
the reward, (it is important to note that crossing the shock or hot plate is entirely up 
to the animal. It can choose not to cross the punishment if it feels the reward is not 
valuable enough.) A common experiment is to balance a less-appealing reward with 
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a more-appealing reward that is given only after a delay. 17 Because animals are usually 
hungry (or thirsty) when running these experiments, they don’t want to wait for reward. 
Thus delay becomes a very simple and measurable punishment to use— how long will the 
animal wait for the reward? 

Of course, humans are animals as well and all of these experiments work well in 
humans: How much effort would a smoker spend to get a puff of a cigarette? 18 How 
much money will you trade for that coffee mug? 19 What negative consequences will you 
put up with to achieve your reward? 20 How long will you wait for that larger reward? 21 
Will you wait for a lesser punishment, knowing that it’s coming? 22 

These four ways of measuring value all come down to quantifiable observations, 
which makes them experimentally (and scientifically) viable. Economists, who study 
humans, typically use hypothetical choices (“What would you do if... ?”) rather than 
real circumstances, which, as we will see later, may not access the same decision-making 
systems. It has been hard to test animals with hypothetical choices (because hypotheti¬ 
cal choices are difficult to construct without language), but there is some evidence that 
chimpanzees with linguistic training will wait longer for rewards described by symbols 
than for immediately presented real rewards. 23 (Remember the chimpanzees and the 
jellybeans?) This, again, suggests that language changes the decision-making machinery. 
More general tests of hypothetical rewards in animals have been limited by our ability to 
train animals to work for tokens. Recent experiments by Daeyeol Lee and his colleagues 
getting monkeys to work for tokens may open up possibilities, but the critical experi¬ 
ments have not yet been done. 24 

One of the most common ways to measure willingness to pay for something is a proce¬ 
dure called th e progressive ratio —the subject (whether it be human or not) has to press a 
lever for a reward, and each time it receives the reward, the number of times it has to press 
the lever for reward increases, often exponentially. The first reward costs one press, the sec¬ 
ond two, the third four, the fourth eight, and so on. Pretty soon, the subject has to press 
the lever a thousand times for one more reward. Eventually, the animal decides that the 
reward isn’t worth that much effort, and the animal stops pressing the lever. This is called 
the break point. Things that seem like they would be more valuable to an animal have higher 
break points than things that seem like they would be less valuable. Hungry animals will 
work harder for food than satiated animals. 25 Cocaine-addicted animals will work harder for 
cocaine than for food. 26 A colleague of mine (Warren Bickel, now at Virginia Tech) told me 
of an experimental (human) subject in one of his experiments who pulled a little lever back 
and forth tens of thousands of times over the course of two hours for one puff of nicotine! 

It’s been known for a long time that drugs are not infinitely valuable. When 
given the choice between drugs and other options, both human addicts and animals 
self-administering drugs A will decrease the amount of drug taken as it gets more expen¬ 
sive relative to the other options. 27 Drugs do show what economists call elasticity: the 
more expensive they get, the less people take. This is why one way to reduce smoking in 
a population is to increase the tax on cigarettes. 


A The animal experimental literature is generally unwilling to call animals “addicts” and instead 
refers to their behavioral actions as “self-administering drugs.” As we will see later in the hook, animals 
self-administer the same drugs that humans do, and animals self-administering drugs show most of the 
same behaviors that humans self-administering drugs do. 



20 


DECISIONS AND THE BRAIN 


Elasticity measures how much the decision to do something decreases in response to 
increases in cost. Luxuries are highly elastic; nonluxuries are highly inelastic. As the costs 
of going to a movie or a ballgame increase, the likelihood that people will go decreases 
quickly. On the other hand, even if the cost of food goes up, people aren’t about to stop buy¬ 
ing food. Some economists have argued that a good definition of addiction is that things 
we are addicted to are inelastic. This allows them to say that the United States is “addicted 
to oil” because we are so highly dependent on driving that our automobile use is generally 
inelastic to the price of oil. However, again, we face an interesting irrationality. The elastic¬ 
ity of automobile use is not linear 28 —raising the price of gasoline from $ 1 per gallon to $2 
had little to no effect on the number of miles driven, but when a gallon of gasoline crossed 
the $3 mark, there was a sudden and dramatic drop in the number of miles driven in 2005. 
People suddenly said, “That’s not worth it anymore.” Of course, people then got used to 
seeing $3 per gallon gasoline and the number of miles driven has begun to increase again. 

This irrationality is where the attempt to measure value gets interesting. In a recent 
set of experiments, Serge Ahmed and his colleagues found that even though the break 
point for cocaine was much higher than for sweetened water,® when given the choice 
between cocaine and sweetened water, almost all of the animals chose the sweetened 
water. 29 (Lest readers think this is something special about cocaine, the same effects 
occur when examining animals self-administering heroin. 30 ) Measuring value by how 
much the animal was willing to pay said that cocaine was more valuable than sweetened 
water. (Cocaine had a higher break point than the sweetened water did.) But measuring 
value by giving the animals a choice said that the sweetened water was more valuable. 
(They consistently chose the sweetened water over the cocaine.) The two measures gave 
completely different results as to which was more valuable. 

One of my favorite examples of dris irrationality is a treatment for addiction called 
Contingency Management, where addicts are offered vouchers to stay off drugs. 31 If addicts 
come into the clinic and provide a clean urine or blood sample, showing that they have 
remained clean for the last few days, they get a voucher for something small—a movie rental 
or a gift certificate. Even very small vouchers can have dramatic effects. What’s interesting 
about this is that raising the cost of a drug by $3 would have very little effect on the amount 
of drug an addict takes (because drugs are inelastic to addicts), but providing a $3 voucher 
option can be enough to keep an addict clean, straight, and sober. Some people have argued 
that this is one of the reasons that Alcoholics Anonymous and its 12-step cousins work. c 
It provides an option (going to meetings and getting praised for staying sober) that works 
like a voucher; it’s a social reward that can provide an alternative to drug-taking. 32 

So why is it so hard to measure value? Why are we so irrational about value? I’m 
going to argue that value is hard to measure because value is not something intrinsic to 
an object. 


B Ahmed and colleagues used saccharin in the water rather than sugar because saccharin has no 
direct nutritive value but tastes sweet to both rats and humans. 

c The Anonymous meetings (AA, Narcotics Anonymous, Gamblers Anonymous, etc.) and their 
12-step cousins also include additional elements that seem to access important decision-making com¬ 
ponents. One example is the presence of a “sponsor” (someone you can call 24/7), who provides a 
low-cost option to pull one away from the high-value drug-taking option. Suicide hotlines also work 
this way, providing a very low-cost option (a phone call) that can short-circuit a dangerous path. 
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First, I’m going to argue in this book that there are multiple decision-making systems, 
each of which has its own method of action-selection, 33 which means there are multiple 
values that can be assigned to any given choice. Second, in the deliberative experiments 
we’ve been looking at, we have to calculate the value of the available options each time. 34 
(Even recognizing that options are available can be a complex process involving mem¬ 
ory and calculation.) The value of a thing depends on your needs, desires, and expecta¬ 
tions, as well as the situation you are in. By framing the question in different ways, we 
can guide people’s attention to different aspects of a question and change their valuation 
of it. In part, this dependence on attention is due to the fact that we don’t know what’s 
important. A minivan can carry the whole family, but the hybrid Prius gets better gas 
mileage, and that convertible BMW looks cooler. It’s hard to compare them. 

This means that determining the true value of something depends on a lot of factors, 
only some of which relate to the thing itself. We’ll see later that value can depend on how 
tired you are, on your emotional state, and on how willing you are to deliberate over the 
available options. 3S It can even be manipulated by changing unrelated cues, such as in 
the anchor effect, in which unrelated numbers (like your address!) can make you more 
likely to converge to a higher or lower value closer to that number. 36 

Economists argue that we should measure value by the reward we expect to get, 
taking into account the probability that we will actually get the reward, and the 
expected investment opportunities and risks. 37 If we tried to explore all possibilities 
and integrate all of this, we could sit forever mulling over possibilities. This leads us 
to the concept of bounded rationality, introduced by Herb Simon in the 1950s, which 
suggests that the calculation takes time, that sometimes it’s better to get to the answer 
quickly by being less complete about the full calculation, and that sometimes a pretty 
good job is good enough. 38 Instead, we use heuristics, little algorithms that work most 
of the time. 

The problem with this theory is that even when we are given enough time, we con¬ 
tinue to use these heuristics. Economists and psychologists who argue for bounded 
rationality (such as Gerd Gigerenzer 39 ) argue that evolution never gives us time. But if 
the issue were one of time, one would expect that the longer one was given, the more 
rational one would be. This isn’t what is seen. People don’t change their minds about 
taking the middle-value television if they are given more time; in fact, they become more 
likely to pick the middle television, not less, with more time. 40 

We pick the middle television in the three-television example because one of the 
algorithms we use when we want to compare the choices immediately available to us 
is to find the middle one. Another example is that we tend to round numbers off by 
recognizing the digits. 41 $3.99 looks like a lot less than $4.00. Think about your “willing¬ 
ness to pay” $3.75 per gallon for gasoline relative to $3.50 per gallon. If both options 
are available, obviously, you’ll go to the station selling it for $3.50. But if it goes up from 
$3.50 to $3.75 from one day to the next, do you really stop buying gasoline? Now, imag¬ 
ine that the price goes up from $3.75 to $4.00. Suddenly, it feels like that gasoline just 
got too expensive. We saw a similar thing at the soda machine by my office. When the 
cost went up from 75 <t to $1, people kept buying sodas. But then one day it went up 
from $ 1 to $ 1.25, and people said, “It’s not worth it” and stopped, (it’s now $ 1.50 and no 
one ever buys sodas there anymore.) Sometimes the cost crosses a line that we simply 
won’t put up with anymore. 
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Part of this is due to the mechanisms by which we categorize things. Whole dollar 
amounts draw our attention and we are more likely to pick them. In a recent experiment, 
Kacey Ballard, Sam McClure, and their colleagues asked people which they would 
prefer, $7 today or $20 in a week. 42 (This is a question called delay-discounting, which 
we will examine in detail later in our chapter on impulsivity [Chapter 5].) But then they 
asked the subjects to decide between $7.03 today and $20 in a week. People were more 
likely to pick the $20 over $7.03 than $20 over $7 even. In a wonderful control, they 
then asked the subjects to decide between $7 today and $20.03 in a week. This time, 
people were more likely to pick the $7. There is no way that these decisions are rational, 
but they do make sense when we realize (1) that making the decision requires determin¬ 
ing the value of the two options anew each time and comparing them, and (2) that we 
use heuristics that prefer even dollar amounts to compare them. D 

Value is an important concept to understanding decision-making, but our brains 
have to calculate how much we value a choice. That calculation is based on heuristics 
and simple algorithms that work pretty well most of the time but can be irrational under 
certain conditions. How do we actually, physically determine value? What are the neu¬ 
rophysiological mechanisms? That brings us to the differences between pleasure, pain, 
and the do-it-again signal. 

Books and papers for further reading 

• Daniel Ariely (2008). Predictably Irrational: The Hidden Forces that Shape Our 
Decisions. New York: HarperCollins. 

• Daniel Kahneman (2011). Thinking, Fast and Slow. New York: Farrar, Straus and 
Giroux. 

• Scott Pious (1993). The Psychology of Judgment and Decision-Making. New York: 
McGraw-Hill. 


D Why people prefer even dollar amounts is not known. Perhaps even dollar amounts are easier to 
calculate with. Perhaps people are suspicious of odd amounts because they are concerned that there’s a 
trick being pulled (like the extra 2k g used by American gas stations). Perhaps people can measure the 
even dollar amounts more easily, which may draw their attention. Whatever the heuristic, people do 
prefer even dollar amounts. 43 
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At the base of dead man's hill, 
sled toppled in a mound of snow, 
exhilaration lights up his face. 
Lets do that again! 


Pleasure and value are different things. One can have pleasure 
without value and value without pleasure. They are signaled by 
different chemicals in the brain. In particular, the brain contains 
a signal for "better than expected," which can be used to learn to 
predict value. The lack of expected reward and the lack of expected 
punishment are separate processes (disappointment, relief) that 
require additional neural mechanisms. 


Euphoria and dysphoria 

It is commonly assumed that pleasure is our interpretation of the things we have to do 
again. “People seek pleasure and avoid pain.” While this is often true, a better descrip¬ 
tion is that people seek things that they recognize will have high value. As we will see 
below, pleasure is dissociable from value. 

Although the saying is “pleasure and pain,” pain is not the opposite of pleasure. Pain 
is actually a sensory system, like any of the other sensory systems (visual, auditory, etc.). 1 
Pain measures damage to tissues and tilings that are likely to damage tissues. It includes 
specific receptors that project up to mid-brain sensory structures that project to cortical 
interpretation areas. The sensation of pain depends on cortical activation in response to 
the pain sensors, but this is true of the other sensory systems as well. The retina in your 
eyes detects photons, but you don’t see photons—you see the objects that are interpreted 
from photons. Pleasure, in contrast, is not a sensation, but an evaluation of a sensation. 

Euphoria, from the Greek word ipopia ( phoria , meaning “bearing” or “feeling”) and 
the Greek root eu- (eu-, meaning “good”), is probably a better word than “pleasure” for 
the brain signal we are discussing. And, of course, euphoria has a clear antonym in dys¬ 
phoria, meaning “discomfort,” deriving from the Greek root Suer- (dys-, bad). The terms 
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euphoria and dysphoria represent calculations in the brain of how good or bad some¬ 
thing is. 


Differences between euphoria and reinforcement 

Watching animals move toward a reward, early psychologists were uncomfortable attrib¬ 
uting emotions to the animal, saying that it “wanted the reward” or that it felt “pleasure” 
at getting the reward. Instead they defined something that made the animal more likely 
to approach it as reinforcement (because it reinforced the animal’s actions). 

Experiments in the 1950s began to identify that there was a difference between 
euphoria and reinforcement in humans as well. In the 1950s, it was found that an elec¬ 
trode placed into a certain area of the brain (called the medial forebrain bundle) would, 
when stimulated, lead to near-perfect reinforcement. 2 Animals with these stimulations 
would forego food, sex, and sleep to continue pressing levers to produce this brain 
stimulation. 

From this, the medial forebrain bundle became popularly known as “the pleasure 
center.” 3 But even in these early experiments, the difference between euphoria and rein¬ 
forcement was becoming clear. Robert Heath and his colleagues implanted several stim¬ 
ulating electrodes into the brain of a patient (for the treatment of epilepsy), and then 
offered the patient different buttons to stimulate each of the electrodes. On stimulation 
from pressing one button, the patient reported orgasmic euphoria. On stimulation from 
the second button, the patient reported mild discomfort. However, the patient contin¬ 
ued pressing the second button over and over again, much more than the first. Euphoria 
and reinforcement are different. 4 

It turns out that the key to the reinforcement from medial forebrain bundle stimula¬ 
tion is a brain neurotransmitter called dopamine. 5 Dopamine is chemically constructed 
out of precursors 23 in a small area of the brain located deep in the base of the midbrain 
called the ventral tegmental area and the substantia nigra. B 


A The pharmacology term “precursors” is used to identify molecules that are converted into the 
molecule being studied. 6 Your cells are, in a sense, chemical factories that convert molecules to other 
molecules, using specialized molecules called enzymes. For example, the chemical levodopa (L-dopa) is 
a precursor for dopamine, which means that if you had extra levodopa in your system, your cells would 
have more building blocks with which to make more dopamine, and you would find that your cells 
had more dopamine to use. This is why levodopa forms a good treatment for Parkinson’s disease, since 
Parkinsons disease is, in fact, a diminishment of dopamine production in certain areas of the brain. 7 

B The ventral tegmental area is so called because it is ventral (near the base of the brain) in the tegmen¬ 
tum (Latin for “covering”), an area that covers the brain stem. Substantia nigra is Latin for “black stuff,” so 
named because it contains melatonin, which makes the area appear black on a histological slice. 

Melatonin is a chemical used by the body to signal the nighttime component of day/night cycles. 8 
This is why it is sometimes used to reset circadian rhythm problems. It interacts with dopamine, par¬ 
ticularly the release of dopamine in the brain, which means that manipulations of melatonin (say for 
jetlag) can affect learning and cognition. This is why pharmacology is so complex: 9 manipulating one 
thing often produces lots of other effects. 
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An early theory of dopamine was that it was the representation of pleasure in the 
brain. 10 Stimulating the medial forebrain bundle makes dopamine cells release their 
dopamine. 11 (Remember that these stimulating electrodes were [incorrectly] hypoth¬ 
esized to be stimulating the “pleasure center.”) In animal studies, blocking dopamine 
blocked reinforcement. 12 Although one could not definitively show that animals felt 
“pleasure,” the assumption was that reinforcement in animals translated to pleasure 
in humans. Most drugs of abuse produce a release of dopamine in the brain. 13 And, of 
course, most drugs of abuse produce euphoria, at least on early use. c Since some of the 
largest pharmacological effects of drugs of abuse are on dopamine, it was assumed that 
dopamine was the “pleasure” signal. This theory turned out to be wrong. 

In a remarkable set of experiments, Kent Berridge at the University of Michigan set 
out to test this theory that dopamine signaled pleasure. 15 He first developed a way of 
measuring euphoria and dysphoria in animals—he watched their facial expressions. 
The idea that animals and humans share facial expressions was first proposed by Charles 
Darwin in his book The Expression of the Emotions in Man and Animals. Berridge and 
his colleagues used cameras tightly focused on the faces of animals, particularly rats, to 
identify that sweet tastes (such as sugar or saccharin) were accompanied by a licking of 
the lips, while bitter tastes (such as quinine) were accompanied by a projection of the 
tongue, matching the classic “yuck” face all parents have seen in their kids being forced 
to eat vegetables. 

What is interesting is that Berridge and his colleagues (particularly Terry Robinson, 
also at the University of Michigan) were able to manipulate these expressions, but not 
with manipulations of dopamine. Manipulations of dopamine changed whether an ani¬ 
mal would work for, learn to look for, or approach a reward, but if the reward was placed 
in the animal’s mouth or even right in front of the animal, it would eat the reward and 
show the same facial expressions. What did change the facial expressions were manipu¬ 
lations of the animal’s opioid system. The opioid system is another set of neurotransmit¬ 
ters in the brain, which we will see are very important to decision-making. They are 
mimicked by opiates—opium, morphine, heroin. Berridge and Robinson suggested that 
there is a distinction between “wanting something” and “liking it.” Dopamine affects the 
wanting, while opioids affect the liking. 


Opioids 

The endogenous opioid system includes three types of opioid signals and recep¬ 
tors in the nucleus accumbens, hypothalamus, amygdala, and other related areas. 16 
Neuroscientists have labeled them by three Greek letters (mu [p], kappa [k], and delta 
[S]). Each of these receptors has a paired endogenous chemical (called a ligand) that 
attaches to it ( endorphins associated with the p-opioid receptors, dynorphin associated 
with the K-opioid receptors, and enkephalins associated with the S-opioid receptors). 
Current work suggests that activation of the p-opioid receptors signals euphoria, and 
activation of the K-opioid receptors signals dysphoria. The functionality of the S-opioid 


c Not all abused drugs are euphoric. Nicotine, for example, is often highly dysphoric on initial use, 
even though it is one of the most reinforcing of all drugs. 14 
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receptors is less clear but seems to be involved in analgesia (the suppression of pain), the 
relief of anxiety, and the induction of craving. Chemicals that stimulate p-opioid recep¬ 
tors (called agonists ) are generally rewarding, reinforcing, and euphorigenic. The classic 
p-opioid receptor agonist is the opium poppy, particularly after it is refined into mor¬ 
phine or heroin, p-opioid receptor agonists are reliably self-administered when given 
the opportunity, whether by humans or other animals. In contrast, chemicals that block 
p-opioid receptors (called antagonists) are aversive and dysphoric and interfere with 
self-administration. On the opposite side, K-opioid receptor agonists are also aversive 
and dysphoric. 

When Kent Berridge and his colleagues manipulated the opioid system in their ani¬ 
mals, they found that they could change the animals’ facial responses to positive (sweet) 
and negative (bitter) things. Stimulating K-opioid receptors when animals were fed the 
sweet solution led to the yuck face, while stimulating p-opioid receptors when animals 
were fed the bitter solution led to licking the lips. This hypothesis is further supported 
by introspective manipulations in humans as well. Naltrexone, a general opioid antago¬ 
nist, has been shown to block the feeling of pleasure on eating sugar—subjects say that 
they can taste the sweetness, but they don’t “care” about it. 17 

If pleasure or euphoria in the brain really is activation of p-opioid receptors, then 
this is a great example of the importance of recognizing our physical nature. Euphoria 
is not a mental construct divorced from the physical neural system—it is instantiated 
as a physical effect in our brains. That means that a chemical that accesses that physi¬ 
cal effect (in this case the p-opioid receptors) can create euphoria directly. This also 
brings up a concept we will return to again and again in this book—the idea of a vulner¬ 
ability or failure mode. We did not evolve p-opioid receptors to take heroin; we evolved 
p-opioid receptors so that we could recognize things in our lives that have value and 
thus give us pleasure. But heroin stimulates the p-opioid receptors directly and pro¬ 
duces feelings of euphoria. Heroin accesses a potential failure mode of our brains—it 
tricks us into feeling euphoria when we shouldn’t. At least, it does so until the body 
gets used to it and dials down the p-opioid receptor, leading to the user needing to 
take more and more heroin to achieve the same euphoria, until eventually even mas¬ 
sive doses of heroin don’t produce euphoria anymore, they just relieve the persistent 
dysphoria left behind. 


Reinforcement 

So if euphoria and dysphoria are the opioid receptors, then what is reinforcement, and 
what is its negative twin, aversion ? According to the original scientific definitions first 
put forward by the animal learning theorists (for example by John Watson, Ivan Pavlov, 
and Edward Thorndike in the first decades of the 20th century, and Clark Hull and 
B. F. Skinner in the 1940s and 1950s), reinforcement is the process by which a reward 
(such as food to a hungry animal) increases responding, while aversion is the process 
by which a penalty decreases responding. 18 (Notice how this definition is defined 
operationally—in terms of directly observable behavioral phenomena. We will break 
open the decision-making mechanism in the next section and move beyond these sim¬ 
ple behavioral limitations. For now, let us be content with observing behavior.) 
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When phrased in terms of reinforcement, the question becomes When does a 
reward reinforce behavior? In the 1960s, Leon Kamin showed that a stimulus reinforced 
a behavior only if another stimulus had not already reinforced it. 19 He called this block¬ 
ing because the first stimulus “blocked” the second. In its simplest form, if you trained 
an animal to push a lever to get a reward in response to a tone and then tried to train 
the animal to respond to both the tone and a light presented together, the light would 
never become associated with the reward. You can test this by asking whether the ani¬ 
mal presses the lever in response to the light alone. It won’t. In contrast, if you had 
started training with both the tone and the light at the same time (that is, if you had 
not pretrained the animal with the tone), then it would have learned to respond to both 
the tone and the light when they were presented together. Learning occurs only when 
something has changed. 

In 1972, Robert Rescorla and Allan Wagner proposed a mathematical theory based 
on the idea that reinforcement was about surprise and predictability. 20 They proposed 
that an animal learned to predict that a reward was coming, and that it learned based on 
the difference between the expectation and the observation. The light was blocked 
by the tone because the tone already completely predicted the reward. This theory is 
supported by more complicated versions of Kamins blocking task, where the reward 
is changed during the tone + light training. 21 If the delivered reward or punishment is 
larger after tone + light than after just the tone, then the animal will learn to associate 
the changed reward with the light. If the delivered reward is smaller after tone + light 
than after just the tone, then the animal will learn that the light is a “negative” predictor. 
Testing this gets complicated—one needs a third cue (say a different sound): train the 
animal to respond to the third cue, then present the third cue with the light, and the ani¬ 
mal responds less because the light predicts that the animal will receive less reward than 
expected from the other cues alone. Learning can also reappear if other things change, 
such as the actual reward given (apples instead of raisins), even when one controls for 
the value of the reward (by ensuring that the animal would work at similar levels for 
both rewards). 

In the 1980s, two computer scientists, Richard Sutton and Andy Barto, laid out a 
new mathematical theory called temporal difference reinforcement learning, which built in 
part on the work of Rescorla and Wagner, but also on earlier work in control theory and 
“operations research” 0 from the 1950s, particularly that of Richard Bellman. 22 Bellman 
originally showed that one can learn an optimal strategy by measuring the difference 
between the observed value and the expected value and learning to take actions to 
reduce that difference. Sutton and Barto recognized the similarity between Bellman’s 
operations research work and the Rescorla-Wagner psychology model. They derived 
an algorithm that was mathematically powerful enough that it could be used to actually 
train robots and other control systems. 


D Control theory uses equations to determine how to change variables in response to stimuli so as 
to have desired effects. The thermostat (negative feedback) that we saw at the very beginning of the 
book is often the first example used in control theory textbooks. Cruise control in your car and the 
stability maintained by current fighter jets are both derived from modern control theory mechanisms. 
Operations research is the term often used for university departments studying these questions. 
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There were two major differences between the Sutton and Barto equations and the 
Rescorla and Wagner theory. First, Rescoria and Wagner assumed that predictions 
about reward were derived from single, individual cues. In contrast, decisions in the 
Sutton and Barto theory were based on categorizations of the situation of the world 
(which they called the “state” of the world). 23;E 

Second, while Rescorla and Wagner’s theory learned to predict reward, Sutton and 
Barto’s learning system learned to predict value. 1 * Each potential situation was associ¬ 
ated with a value. Actions of the agent (the computer program or the animal) and events 
in the world could change the situation. Once the expected value of each situation was 
known, the agent could decide the best thing to do (take the action leading to the situ¬ 
ation with the highest expected value). The key was that Sutton and Barto showed that 
you could learn the value function (what the value of each situation was) at the same 
time that you were taking actions. F 

Sutton and Barto called this the actor-critic architecture, because it required two com¬ 
ponents—an actor to take actions based on the current estimation of the value function 
and a critic to compare the current estimation of the value function with the observed 
outcomes. The critic calculated the difference between the value you actually got and 
the value you expected to get, which Sutton and Barto called value-prediction error and 
labeled as delta (§), from the analogy to the common physics and mathematical use of 
the term A (delta) meaning “difference.” 13 


Value-prediction error 


The concept of value-prediction error (S) is easiest to understand by an example. 
Imagine that there is a soda machine in the hallway outside your office, and the soda 
is supposed to cost $1. You put your dollar in the machine, and you get two sodas out 


E I prefer to use the term situation rather than the typical computer science term state (or the term 
preferred by psychology —stimulus set). State is used in many other literatures to refer to things like 
“internal state” (such as being under the influence of a drug, or not being under the influence of the 
drug), so overloaded terms like state can be confusing. Although many animal psychology experiments 
do use a single stimulus, look around you—what is the stimulus that is driving your behavior? I am 
currently listening to music and watching the sun rise above the snow outside my house, my feet are 
in fuzzy slippers, but my ankles are chilly—which of these stimuli are important? As we will see in 
Chapter 12, how we identify the important stimuli is a critical component of the decision-making sys¬ 
tem, so simply saying stimulus is incomplete. However, conveniently, state, stimulus, and situation all 
start with S, so if we wanted to write equations (as scientists are wont to do), we don’t have to change 
our variables. 

F There is a tradeoff between exploration and exploitation. If you take only the best choice, you 
might miss opportunities. If you go exploring, you might make mistakes. For our discussion here, we 
will assume that there is enough exploration not to get trapped in a situation where you know a good 
answer but haven’t found the best one yet. We will explore the exploration-exploitation tradeoff in 
depth in its own chapter (Chapter 14). 

G One of the problems with the science of decision-making is that it draws from lots of fields, each 
of which has its own terminology. The 8 used in computer science to represent the value-prediction 
error is unrelated to the 8 used to differentiate types of opioid receptors. 



Value , Euphoria, and the Do-It-Again Signal 


29 


instead of one. Whatever the reason, this result was better than expected (8, the differ¬ 
ence between what you observed and what you expected, is positive) and you are likely 
to increase your willingness to put money into this machine. If, in contrast, you put your 
dollar in and get nothing out, then 8 is negative and you are likely to decrease your willing¬ 
ness to put money into this machine. If, as expected, you put your dollar in and you get 
your soda out, your observations match your expectations, 8 is zero, and you don’t need 
to learn anything about this soda machine. Notice that you still get the nutrients (such 
as they are) from the soda. You still get the pleasure of drinking the soda. You still know 
what actions to take to get a soda from the soda machine. But you don’t need to learn 
anything more about the machine. 

In what I still feel is one of the most remarkable discoveries in neuroscience, in the 
early 1990s, Wolfram Schultz and his colleagues found that the transient bursts of firing 
in dopamine neurons increase with unexpected increases in value, decrease with unex¬ 
pected decreases in value, and generally track the prediction-error signals proposed 
by Rescorla and Wagner and by Sutton and Barto. 25 At the time (the late 1980s and 
early 1990s), dopamine was primarily known as a reward or pleasure signal (remember, 
this is wrong!) in the animal behavior literature and as a motor-enabling signal in the 
Parkinson’s disease literature. H 

Schultz was looking at dopamine in animals learning to make associations of 
stimuli to reward under the expectation that he would then go on to look at its role in 
Parkinson’s disease. The first step was to examine the firing of dopamine cells in response 


H Parkinson’s disease was first identified in the 1800s by James Parkinson, who wrote a monograph 
called An Essay on the Shaking Palsy. It is not uncommon in modern societies, but descriptions consis¬ 
tent with Parkinson’s go back to ancient times. 26 For example, a description exists in Indian Ayurvedic 
texts that is likely Parkinson’s disease, and it was treated with a medicinal plant that is now known 
to contain levodopa (a common modern treatment for Parkinson’s disease). Throughout most of the 
20th century, Parkinson’s disease has been thought of as a dysfunction in the motor system—most 
patients with Parkinson’s show an inability to initiate movement (akinesia) or a slowing of the initiation 
of movement (bradykinesia). 27 For example, patients with the disease develop posture problems and 
tremors in their limbs both before and during movement. Parkinson’s disease, however, is not a disor¬ 
der of the muscles or the actual motor system—there are situations where a Parkinson’s patient can 
react surprisingly well. 28 This is nicely shown in the movi e Awakenings, where Rose is able to walk to the 
window once the floor tiles are painted black and white. We will see later in this book that these are due 
to the intact nature of other decision-making systems in Parkinson’s patients. Whether these are due 
to emotional reactions (running from a room during a fire, Chapter 8), to simple, long-stored reflexive 
action sequences (surprised by a thrown ball and “Catch!,” Chapter 10), or to the presence of explicit 
visual cues (lines on the floor enabling access of Deliberative systems, Chapter 9) is still unknown. 29 

Neurophysiologically, Parkinson’s disease is due to the loss of dopamine neurons. 30 How the effects 
of dopamine loss in Parkinson’s disease are related to the issues of dopamine as a 8 signal is complex. 
The 8 signal observed by Schultz and colleagues is related to fast changes in dopamine firing (bursts, 
called phasic signaling). Dopamine cells also fire at a regular baseline rate (called tonic firing). 31 The 
relationship between the phasic bursts, the tonic firing, Parkinson’s disease, and decision-making is 
still unknown, but it is known that decision-making is impaired in Parkinson’s disease, 32 and several 
hypotheses have been proposed. 33 Tonic levels of dopamine have been suggested to play roles in the 
recognition of situations 34 (this chapter, below, and Chapter 12) and in the invigoration of movement 35 
(motivation, Chapter 13). 
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to a reward. In these first experiments, Schultz and his colleagues found that dopamine 
neurons would respond whenever the thirsty animal received a juice reward. But in this 
first experiment, the reward was just delivered at random times; when Schultz and his 
colleagues provided a cue that predicted the reward, to their surprise, the dopamine 
neurons no longer responded to the reward. Instead, the cells responded to the cue that 
predicted the reward. 36 

These experimental results were unified with the theoretical work by Read Montague, 
Peter Dayan, and Terry Sejnowski, who showed that the unexpected-reward neurons 
reported by Wolfram Schultz and his colleagues were exactly the “delta” S signal needed 
by the Sutton and Barto temporal difference learning algorithm. Over three years (1995, 
1996, and 1997), they published three papers suggesting that the monkey dopamine 
cell data matched the value-prediction error signals needed for the temporal difference 
reinforcement learning theory. 37 

To really understand how dopamine tracks the delta signal, one can ask Why do the 
dopamine neurons increase their responding to the cue? Before the cue, the animal has no 
immediate expectation of reward. (For all it knows, the days session is about to end.) It 
is in a low-value situation. When the cue is shown to the animal, it realizes that it is going 
to get a reward soon, so it is now in a higher-value situation. The change from low-value 
“waiting” to high-value “going to get reward” is an unexpected change in value and pro¬ 
duces a delta signal of “better than expected.” If the animal has learned that an instruction 
stimulus predicts the cue, saying that a cue will be arriving soon, then the dopamine sig¬ 
nal will appear at the earlier instruction signal rather than at the cue (because the instruc¬ 
tion signal is now the unexpected increase in value). If the cue then doesn’t arrive, you get 
a decrease in dopamine because the cue was expected but not delivered and you have an 
unexpected decrease in value. 

Subsequent experiments have found that the signal does track the S value- 
prediction error signal remarkably well. If the reward was larger than the cue pre¬ 
dicted, dopamine neurons increase responding again, while if the reward was not 
delivered when expected, the dopamine neurons decrease responding. Dopamine 
neurons have a small baseline firing rate of a few spikes per second. Positive 
responses entail a burst of spikes, while negative responses entail a cessation of 
spikes. 38 This allows dopamine neurons to signal both “better-than-expected” and 
“worse-than-expected.” Dopamine even tracks the value-prediction error signals in 
Kamin blocking paradigms. 39 These results have since been replicated in rats, mon¬ 
keys, and humans. 40 


Reinforcement and aversion, disappointment and relief 

In the computer models, penalties are generally written as negative rewards 
and aversion is written as negative reinforcement. However, the examples of 
“worse-than-expected” used in the experimental data described above are not actu¬ 
ally punishments, nor do they produce actual aversion. Rather, they are examples of 
disappointment —a lack of delivery of an expected reward. 41 When researchers have 
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examined actual aversive events, the dopamine response to those events has turned 
out to be very complicated. 42,1 

As we will see below, aversion and disappointment are actually different phenom¬ 
ena and cannot work by a shared mechanism. Some researchers have suggested that 
there is another neurotransmitter signal analogous to dopamine that can serve as the 
aversion signal (in parallel to the reinforcement signal of dopamine), 47 but no one has 
(as yet) found itJ Responses to punishments (like retreating from painful stimuli) 
entail a different neural circuitry than responses to positive rewards, in part because the 
responses to punishment are an evolutionarily older circuit. 54 Learning not to do some¬ 
thing (aversion) uses a different set of neural systems than learning to do something 
(reinforcement) , 55 


1 Just how well the value-prediction error (8) theory explains the dopamine data is a point of much 
controversy today. Part of the problem is that all experiments require behavior, mammalian behavior is 
a complex process, and we often need to run computer simulations of the behavior itself to determine 
what a value-prediction error signal would look like in a given behavioral experiment. 

For example, Sham Kakade and Peter Dayan showed that if two signals, one positive and one neu¬ 
tral, are provided to the agent, and the sensory cues from the two signals are similar, then one can see 
“generalization effects” where there is an illusory positive 8 signal to the neutral stimulus that is fol¬ 
lowed quickly by a less-than-expected negative 8 signal. 43 Imagine two doors, one that leads to food and 
one that doesn’t. At the sound of a door opening, you might look to the two doors, getting a positive 
8 signal, but then when you realize that the neutral (nonrewarded) door was the one that was opened, 
you would be disappointed (giving you a negative 8). 

As another example, Patryk Laurent has shown that an agent with limited sensory resources is evo¬ 
lutionarily well served by a positive orienting signal, even to potentially aversive events. 44 Essentially, 
this orienting signal allows the agent to do better than expected at avoiding those aversive events. 

Both of these examples would provide complex dopamine signals to aversive events, explaining 
some of the dopamine signals seen to neutral and aversive events. However, they make specific predic¬ 
tions about how dopamine should respond to those events. Schultz and colleagues have found that 
the predicted less-than-expected reaction after generalization dopamine signals occurs 45 Although the 
Laurent hypothesis has not been explicitly tested, it may explain the very fast response signals seen by 
Peter Redgrave and colleagues. 46 

J Some researchers have suggested that serotonin might play the role of the negative (aversive) 
signal, 48 but recent experiments have shown that not to be the case. 49 Another interesting potential 
candidate is norepinephrine (called noradrenalin in Europe). Norepinephrine is a chemical modifica¬ 
tion of dopamine, making dopamine a precursor to it. 50 In invertebrates, the norepinephrine analogue 
octopamine serves as the positive error signal, while dopamine serves as the negative error signal. 51 All 
of these molecules (serotonin, norepinephrine, octopamine, and dopamine) are very similar in their 
molecular structure, 52 making them plausible candidates, given the copy-and-modify process often 
seen in evolution. 

Additionally, some recent very exciting work has found that neurons in another neural struc¬ 
ture, the habenula (which projects to the dopamine neurons in the ventral tegmental area), increase 
their firing with aversion and decrease their firing with reinforcement. 53 It is still unclear whether 
these habenula neurons are the long-sought aversion neurons or if they are part of the brains calcula¬ 
tion of 8. Remember, 8 is a mathematical term. Dopamine reflects 8, therefore, the brain calculates 
8... somehow. 
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Figure 4.1 Providing a positive thing (a reward, leading to euphoria ) produces 
reinforcement, while the lack of delivery of an expected reward leads to disappointment. 
Similarly providing a negative thing (a punishment, leading to dysphoria ) produces 
aversion, while the lack of delivery of an expected punishment leads to relief. 


From the very beginning of animal learning experiments, it has been known that 
the process of recognizing that a cue no longer predicts reward does not mean that 
the animal has forgotten the association. 56 This goes all the way back to Pavlov and his 
famous dogs learning to associate bells with food. Once the dogs had learned to salivate 
in response to the bell (in anticipation of the food), when the food was not presented 
to them, they showed frustration and anger, and they searched for the food. Eventually, 
however, they recognized that the bell no longer implied food and learned to ignore the 
bell. If, however, they were given food again just once after the bell was rung, they imme¬ 
diately started salivating for food the next time. Because learning was much faster after¬ 
ward than before, Pavlov knew that the dog had not forgotten the original association. 

This means that there must be a second process. In the literature, this process is usually 
called extinction because the lack of delivered reward has “extinguished” the response. 16 
In fact, the exact same effects occur with aversion. If an animal is trained to associate a 
cue with a negative event (say an electric shock), and this association is extinguished 
(by providing the cue without the shock), then it can be reminded of the association 
(one pair of cue + shock) and the response will reappear without additional training. 
Disappointment is not the forgetting of reinforcement and relief is not the forgetting of 
aversion. Because both reinforcement and aversion have corresponding secondary pro¬ 
cesses (disappointment and relief), disappointment cannot be the same process as aver¬ 
sion and relief cannot be the same process as reinforcement. See Figure 4.1. 


Neural mechanisms of extinction 

As this point, scientists are not completely sure what the neural mechanisms are that 
underlie the secondary process of extinction that follows from disappointment or relief. 
There are two prominent theories to explain this secondary process. One is that there 
is a specific second inhibition process, such that neurons in this secondary area inhibit 
neurons in the primary association area. 57 It is known, for example, that synapses in 
the amygdala are critical to the initial learning of an aversive association (lesions of the 
amygdala prevent such associations, the creation of such associations changes firing pat¬ 
terns in the amygdala, and the creation of such associations occurs with and depends on 
synaptic plasticity in the amygdala). However, neurons in the central prefrontal cortex 
(the infralimbic cortex in the rat, the central anterior cingulate cortex in the human) 


K Using the word extinction for this process is unfortunate because it has no relation to the evolu¬ 
tionary term extinction, meaning the ending of a species. 
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start firing after extinction and project to inhibitory neurons in the amygdala, which 
stop the firing of the original association cells. Yadin Dudai and his colleagues recently 
found that in humans, these areas are active when people with a phobia of snakes show 
courage and override their fear to perform a task in which they pull the snake toward 
themselves. 58 However, this theory implies that there is only one level of recursion— 
that one can only have an association and a stopping of that association. 

The other theory starts from the assumption that you are not provided a definition 
of the situation you are in, but instead that you have to infer that situation from the 
available cues. 59 Look around you: there are millions of cues. How many of them are 
important to your ability to learn from this book? Around me right now, I am looking 
at windows, trees, an empty coffee cup, squirrels chasing each other. There is the sound 
of birds, a chickadee, a crow. There is the rumble of cars in the distance. There is the feel 
of the chair I am sitting in, the heat of my laptop (which gets uncomfortably hot when I 
work on it too long). Which of these cues are important? Most animal experiments are 
done in very limited environments with few obvious cues except for the tones and lights 
that are to be associated with the rewards and penalties. Animals didn’t evolve to live in 
these empty environments. 

This second theory says that before one can associate a situation with a reward or 
a penalty, one needs to define the set of stimuli that will identify that situation. Any 
situation definition will attend to some cues that are present in the environment and 
ignore others. We (myself and my colleagues Adam Johnson, Steve Jensen, and Zeb 
Kurth-Nelson) called this process “situation-recognition” and suggested that disap¬ 
pointment and relief produce a narrowing of the definition of the situation such that 
the agent would begin to pay attention to additional cues and begin to differentiate the 
new situation from the old one. 60 When animals no longer receive rewards or penalties 
(that is, they are disappointed or relieved), they begin “searching for the cue” that dif¬ 
ferentiates the two situations. 61 Imagine the soda machine we talked about when we 
first encountered the value-prediction error S signal. If you put your money in and get 
nothing out, you aren’t going to forget that putting money in soda machines can get you 
soda; you are going to look for what’s different about this machine. Maybe there’s a light 
that says “out of order” or “out of stock.” Maybe the machine is off. Once you identify 
the difference, your definition of the soda-machine-available situation has changed, and 
you can go find a soda machine that works. 

The second theory suggests that the additional signals being provided from the pre¬ 
frontal cortex are providing additional dimensions on which to categorize the situation. 
It provides explanations for a number of specific phenomena seen in the extinction lit¬ 
erature. For example, there is extensive evidence that extinction is about the recogni¬ 
tion of a change. 62 Animals show much slower extinction after a variable (probabilistic) 
reward-delivery contingency than after a regular (being sure of always getting reward) 
reward contingency. If an animal is provided with a reward only half the time after the 
cue, it will still learn to respond, but it will be slower to stop when the reward is no 
longer delivered. On the other hand, if it is always provided with a reward, then it will 
quickly stop responding when the reward is no longer delivered. However, this is not 
simply a question of counting, because, as shown in Figure 4.2, if there is a pattern, then 
even with only a 50/50 chance of getting a reward, one can easily determine the pattern, 
and animals stop responding as soon as the pattern is disrupted. 
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Figure 4.2 Sequences of reward and punishment. Imagine you are a subject in an 
experiment faced with one of these sequences of reward delivery. (R means you get the 
reward and N means you don’t.) Imagine seeing the sequence from left to right. (Trace 
your finger across to provide yourself a sequence.) When would you stop responding? 

Try each line separately. You will probably find yourself trying to predict whether you get 
a reward or not. When the observations no longer match your predictions, that’s when you 
recognize a change. 


As we saw in Chapter 3, “value” is a complex thing. We have seen in this chapter how 
the calculation of value and the learning to predict value depend on multiple, interact¬ 
ing systems. There is a system measuring the pleasure of a thing (euphoria), a system 
learning the usefulness of repeating actions (reinforcement), and a system recognizing 
change (disappointment). In parallel, there are the dysphoria, aversion, and relief sys¬ 
tems. As we will see in the next chapters, these value-learning components form the 
bases of multiple decision-making systems that interact to drive our actions. The fact 
that there are multiple systems interacting makes the detection and actual definition of 
value difficult. 
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Risk and Reward 


“Carpe diem,” they say, 
but you don’t get a day, 
you get a moment, 
a moment to hesitate.... 
or to act. 


Defining decision-making in terms of action-selection creates the 
potential for decisions that are illogical from a value-based per¬ 
spective. The mammalian decision-making system includes mul¬ 
tiple decision-making components, which can produce conflict 
between action options. 


A teenager floors the accelerator as he navigates the winding curves of a wooded road; the 
car skids at each turn, barely containing its acceleration. A college student spends thou¬ 
sands of dollars on clothes and doesn’t have enough money to pay for rent. An alcoholic 
stops by the bar on the way home. “Just one drink,” he says. Hours later, his kids come to 
take him home. A firefighter stands frozen a moment before a burning building, but then 
gathers her courage and rushes into the building to pull a child to safety. Each of these 
examples is a commonly cited case of a conflict between decision-making systems. 

The teenager is balancing the thrill of speed with the risk of losing any potential future 
he might have. The college student spends the money now for the short-term gain of having 
clothes and ignores (or forgets) the long-term gain of actually staying in her apartment. 
The alcoholic risks his family, his happiness, and often his job and livelihood for the simple, 
immediate pleasures of a drink. (Of course, the problem is that the one drink turns into two 
and then three and then more.) The firefighter must overcome emotional fear reactions that 
say “don’t go in there.” These are all conflicts between decision-making systems. 

Risk-seeking behavior 

Colloquially, risk entails the potential for loss in the face of potential gains. (Our 
hypothetical teenager is risking his life driving too fast.) In economics, the term risk 
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is defined as the variability in the outcome 1 —the stock market has more risk than an 
FDIC-insured savings account. This is a form of uncertainty and it is now known that 
there are representations of risk and uncertainty in ourbrains. 2 As we will discuss in depth 
in Chapter 14, there are differences between expected variability (a known probability, 
such as the chance of rain), unexpected variability (a surprise, such as when something 
that has been working suddenly breaks), and unknown variability (ambiguity, when you 
know that you don’t know the probabilities). Animals will expend energy to translate 
unknown variability into expected variability and to reduce expected variability. 3 

Imagine you are an animal, entering a new environment. You don’t know what dangers 
lurk out there. You don’t know what rewards are available. Before you can make yourself safe 
from those dangers and before you can gain those rewards, you have to explore the environ¬ 
ment. 4 Imagine that you have explored part of the environment. Now you have a choice: Do 
you stay and use the rewards you’ve already found, or do you go looking for better ones? 

Animals tend to be risk-averse with gains and risk-seeking with losses. This is the 
source of the difference in the Asian flu example in Chapter 3—phrased in terms of 
saving lives, people take the safe option, but phrased in terms of losses, people take the 
risky option. 5 But this isn’t the whole story either, because risk and reward interact with 
exploration and uncertainty. 

Interestingly, the most risk-seeking behavior tends to occur in adolescents. 6 
Adolescents are at that transition from dependence to independence. Humans first 
develop through a youth stage during which they are protected and trained. At the sub¬ 
sequent adolescent stage, humans go exploring to see what the parameters of the world 
are. As they age out of adolescence, humans tend to settle down and be less risk-seeking. 
This ability to learn the parameters of the world after a long training period makes 
humans particularly flexible in their ability to interact with the world. 

An interesting question is then to ask What drives exploration? Animals, including 
humans, have an innate curiosity that drives us to explore the world. But as the proverb 
says, “curiosity killed the cat.” Exploring is dangerous. A fair assessment of the dangers 
could lead us to be unwilling to take a chance. On average, however, people tend to 
underestimate dangers and to be overoptimistic about unknowns, 7 which tends to drive 
exploration, while still maintaining intelligent behavior in the parts of the world we do 
know. (Remember, “the grass is always greener on the other side of the hill”—except 
when you get there, and it’s not.) 

So how is the teenager speeding down a road an example of exploration overwhelming 
exploitation? Our hypothetical teenager is unlikely to be speeding down an untraveled road. 
When animals are exploring new territory, they tend to travel very slowly, observing every¬ 
thing around them. 8 But our hypothetical teenager is testing his limits, trying to figure out 
how good his reaction times are, how fast he can go while still maintaining control of the car. 
If the teenager is overestimating his abilities or underestimating the likelihood of meeting an 
oncoming car on that windy, dark road that late at night, the consequences can be tragic. 

We will see throughout this book that personality entails differences in the parame¬ 
ters underlying the decision-making system. Here we have two parameters and an inter¬ 
action: How curious are you? How overoptimistic in the face of the unknown? How willing 
are you to risk danger for those new answers? (Adolescents, of course, are notorious for 
both of these properties—risk-taking behavior driven by curiosity and overoptimism.) 
But each individual, even in both youth and adulthood, has a different take on these 
parameters, 9 and thus each of us has our own level of risk-seeking and overoptimism, 
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our own threshold of what risks we will tolerate, and how badly we estimate those risks. 
Balancing risk and reward can be the difference between success and failure, and finding 
that right balance is not always easy. 


Waiting for a reward 

Which would you rather have, $10 today or $10 in a week? Barring some really strange 
situations, pretty much everyone would want the $10 today. What this means is that $10 
in a week is worth less to you than $10 today. We say that rewards delivered in the future 
are discounted. 10 Logically, discounting future rewards makes sense because things can 
happen between now and then; waiting for the future is risky. 11 If you get hit by a bus or 
win the lottery or the world explodes in thermonuclear war, that $ 10 in a week just isn’t 
worth as much as it was today. 

$10 today can also be invested, 12 so in a week, you could have more than $10. This 
of course works with nonmonetary rewards as well (food, mates, etc.) 13 To a starving 
animal, food now is worth a lot more than food in a week. If you starve to death now, 
food next week just isn’t that valuable. And, of course, in the long term, food now will 
lead to a stronger body, more energy, and more ability to find food later. 

So rewards now are worth more than rewards in the future. By asking people a series 
of these money-now-or-money-later questions ( Which would you rather have, $9 now or 
$10 next week?), we can determine how valuable $10 in a week actually is. If we were to 
plot this as a function of the delay, we can derive a discounting curve and measure how 
value decreases with delay. 14 

Of course, we can’t ask animals this question, 21 but we can offer them a choice 
between a small amount of food now and a large amount of food later. ls For example, in 
a typical experiment, a rat or monkey or pigeon is offered two options (levers 16 or paths 
leading to a food location 17 ): taking the first option provides one food pellet immedi¬ 
ately, but taking the second option provides more (say three) food pellets after a delay. 
By observing the animal’s choices, we can again determine the discounting curve. 

If discounting were simply due to inflation or the ability to invest money, we would 
expect it to follow an exponential function of time. 18 In an exponential function, the value 
decreases by the same percentage every unit of time.® So if your discounting function 
were an exponential with a half-life of a year, you would be equally happy with $5 now 
and $10 in a year or $2.50 now and $10 in two years. Exponential discounting curves 


A We can ask them. But they are unlikely to give us a meaningful answer. 

B Examples of exponential functions include bank accounts and population growth. When a bank 
says that it will pay you a percentage interest each year, the total amount of money in your account will 
increase by that percentage each year. Since the money goes back into your account, you are effectively 
multiplying your savings account by a number greater than 1 each year. Even a small number can build 
up quickly: a 5% interest rate will double in 15 years. Here, we are talking about exponential decay, 
which is like multiplying the value by a number a little less than 1 each year. 

A good example of exponential decay is the decay of radioactive material: after a given number of 
years, half the material will have decayed. This is where the term half-life comes from. For example, 
plutonium-238 has a half-life of 88 years. So if you have a pound of plutonium-238, after 88 years, 
you would have half a pound. After another 88 years (176 years total), you would have a quarter of a 
pound, etc. 
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have a very useful property which is that they are self-similar: your choices now are the 
same choices you would make in a year. 

But neither humans answering questionnaires nor animals pressing levers show 
exponential discounting curves. 19 In fact, humans answering questionnaires, animals 
pressing levers for food, and humans pressing levers for food all show a discounting 
curve best described as a hyperbolic function, which drops off very quickly but then 
becomes more flat with time. 

Any nonexponential discounting curve (including hyperbolic ones) will show a prop¬ 
erty called preference reversal, in which the choices change depending on your temporal 
vantage point. 20 You can convince yourself that you too show preference reversal. Ask 
yourself which you would rather have, $10 today or $11 in a week? Then ask yourself 
which you would rather have, $10 in a year or $11 in a year and a week? Each person 
has a different reversal point, so you may have to find a slightly different set of numbers 
for yourself, but you will almost certainly be able to find a pair of numbers such that you 
won’t wait now, but will wait next year. Notice that this doesn’t make any sense ratio¬ 
nally. Today, you say it’s not worth waiting for that extra dollar, but for a year from now, 
when faced with what is really the same choice, you say you’ll wait. Presumably, if you ask 
yourself next year, you’ll say it’s not worth waiting—you’ll have changed your mind and 
reversed your preferences. Even though this is irrational, it feels right to most people. 

This is the mistake being made by our hypothetical extravagant student, who is say¬ 
ing she would prefer to spend the money now on the immediately available reward (new 
clothes) rather than waiting for the larger reward (paying rent) later. In fact, when her 
parents ask her at the beginning of the school year whether she’s going to pay her rent or 
buy new clothes, she’s likely to tell them (we ll assume honestly) that she’s going to pay 
the rent. This switch (between wanting to pay the rent early and buying clothes) is an 
example of the preference reversal that we’re talking about. 

Preference reversal can also be seen in the opposite direction. This might be called the 
Cathy effect, after the comic strip Cathy by Cathy Guisewite, which has a regular motif 
of Cathy’s inability to resist high-sugar c treats like Halloween candy or birthday cake, 
(in panel 1, Cathy says she won’t eat the cake this time, but in panel 3, when faced with the 
actual cake, we find her eating it.) From a distance, the chocolate is discounted and has low 
value and the long-term value of keeping to her diet is preferred, but when the chocolate is 
immediately available, it is discounted less, and poor Cathy makes the wrong choice. 

In part, Cathy’s dilemma comes from an interaction between decision-making 
systems: from a distance, one system wants to stay on her diet, but from close up, 
another system reaches for the cake. This is the Parable of the Jellybeans that we saw in 
Chapter 1—it is hard to reject the physical reward in front of you for an abstract reward 
later. 22 One possible explanation for the success of Contingency Management (which 
offers concrete rewards to addicts for staying clean of drugs) in addiction and behavioral 
modification is that it offers a concrete alternative option, which allows the subject to 
attend to other options rather than the drug. 23 

In a positive light, preference reversal also allows for an interesting phenomenon 
called precommitment, in which a person or an animal sets things up so that it is not 


c Processed sugar is far sweeter than most foods that we have evolved to eat, so it is a substance that 
overstimulates sensory systems, making intake particularly difficult to control. 21 
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given the choice in the future. 24 For example, an alcoholic who knows that he will drink 
if he goes to the bar can decide not to drive by the bar in the first place. From a distance, 
he makes one choice, knowing that he will make the wrong choice if given an opportu¬ 
nity later. If our hypothetical extravagant student is smart and knows herself well, then 
she might put the money away in a prepaid escrow account that can be used only for 
rent. The two selves (the student at the beginning of the school year who wants to pay 
rent and the student midyear who wants to buy clothes) are in conflict with each other. 25 
If she’s smart, she can precommit herself when she’s that early, thrifty student to prevent 
herself from wasting the money when she’s that later, extravagant student. As we explore 
the actual components of the decision-making system, we will find that precommitment 
involves an interaction among Deliberative (Chapter 9) and Self-Control (Chapter 15) 
systems. 26 The ability to precommit to one option over another remains one of the most 
powerful tools in our decision-making arsenal. 27 


Stopping a prepotent action 

The interaction between multiple decision systems can also produce conflict directly. 
Only some of the multiple decision-making systems that drive actions are reflected in 
our consideration of consciousness. This means that there can be actions taken, after 
which we find ourselves saying, “Now why did I do that?” 

Other examples of conflict (such as the firefighter) come from one system wanting to 
approach or flee a stimulus and another component telling it to hold still. 28 In Chapter 6, 
we will see that there are emotional components that depend on the amygdala (a deep, lim¬ 
bic brain structure involved in simple approach-and-retreat cue-response phenomena 29 ), 
learned-action components that depend on the basal ganglia (other neural structures that 
learn situation-action pairs 30 ), and deliberative components that depend on the prefron¬ 
tal cortex and the hippocampus (cognitive structures that enable the imagination of future 
possibilities 31 ). When these three systems select different actions, we have a conflict. 

Classic experiments on rats and other animals have studied a simple process called 
fear conditioning. 31 In a typical experiment, a rat would be placed in a box, and, at irregu¬ 
lar intervals, a tone would be played and then the rat would receive a mild shock. D In 
modern experiments, the shock is never enough to hurt the animal, just enough to be 
unpleasant. Similar experiments can be done on humans, where they get a mild shock 
after a cue. 33 What happens is that the rat begins to fear the tone that predicts that a shock 
is coming. Humans explicitly describe dreading the oncoming shock and describe how 
unpleasant the expectation of a shock is. In fact, Greg Berns and his colleagues found 
that the dread of waiting for a shock was so unpleasant that people would choose to have 


D It is important to note that both the animal and human experiments are highly regulated. To 
ensure that these experiments are humane and safe, the scientists have to get permission from the 
university’s IRB (internal review hoard, for human experiments) or the university’s IACUC (internal 
animal care and use committee, for animal experiments). Both of these boards include scientists, medi¬ 
cal professionals (vets for animals, MDs for humans), and local community members (often including 
animal rights activists on IACUC hoards). If the committee doesn’t think the experiment is humane, 
safe, and likely to he important scientifically, then the experiment can’t go forward. 
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a larger shock quickly rather than have to wait for a smaller one later. 34;E Rats cannot, 
of course, tell us that they are fearful and nervous about the shock, but they freeze in 
fear and startle more in response to an unexpected event. If you are tense, waiting for a 
shock, then a loud noise will make you jump. Rats do the same thing. 

This brings us to tone-fear conditioning, in which a rat learns to expect a shock after 
a tone is played. Fear conditioning can then be extinguished by subsequently providing 
the rat with the tone but no shock. As we saw in the previous chapter, behavioral extinc¬ 
tion entails new activity in the infralimbic cortex that projects down to the amygdala 
and inhibits the learned fear response. 35 

This matches the Yadin Dudai study on courage that was mentioned in the previous 
chapter—Dudai and his colleagues studied humans overcoming their fear of snakes. 36 
The sight of the snake triggered the emotional (Pavlovian) system that had an innate 
reaction to the snake (“Run away!”) while the Deliberative system had to overcome 
that prepotent action. As is said in many a movie and many a novel, courage is not the 
lack of fear, but the ability to overcome that fear. F The fact that we now know the neural 
mechanisms underlying courage does not diminish what courage is. 


Summary 

In this first part of the book, we’ve defined the question of decision-making in terms of 
action-selection, identified that we cannot simply say that we are evaluating choices, and 
identified that there are multiple systems that drive action-selection within the mammal. 
In the next part, I want to turn to the question of what these multiple decision-making 
systems are and how they interact with each other. Afterwards (in the third part of 
the book), we’ll turn to their vulnerabilities and what those vulnerabilities mean for 
decision-making problems such as addiction and post-traumatic stress disorder. 

Books and papers for further reading 

• George Ainslie (2001). Breakdown of Will. Cambridge, UK: Cambridge University Press. 
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University Press. 

• Uri Nili, Hagar Goldberg, Abraham Weizman, and Yadin Dudai (2010). Fear Thou 
Not: Activity of frontal and temporal circuits in moments of real-life courage. Neuron , 
66, 949-962. 


E I had a similar experience with my fish tank a few months ago, where one of the underwater 
pumps was giving me a shock whenever I put my hand in the water. It became a very definite effort of 
will to put my hand in the water to retrieve the damaged pump. 

F In his epic novel Dune, Frank Herbert defines humanity as the ability to hold your hand in a pain- 
box, refusing the Reflexive and Pavlovian action-selection systems that want to pull the hand away, 
because the Deliberative system knows that the woman testing you will kill you with her poison pin at 
your neck if you do. 37 
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Miles’ trumpet 
syncopates the drumbeat, 
rises like bubbling voices 
out of a turbid pool, 
floats like soap bubbles lost in air, 
fading into the morning mists. 


There are multiple neural systems that drive decision-making, 
each of which has different computational properties that make 
it better suited to drive action-selection under different conditions. 
We will identify four action-selection systems: Reflexes, Pavlovian, 
Deliberative, and Procedural. Along with motoric, perceptual, 
situation-categorization, and motivational support systems, these 
make up the decision-making system. 


Imagine the first time you drive to work. You would probably look at a map, plan the best 
route. During the drive, you would pay particular attention to road signs and landmarks. 
But driving that same route each day for weeks, months, or years, the route becomes 
automatic. It no longer requires the same amount of attention, and you can think about 
other things: the test you have to write, your kids’ soccer game that evening, what you’re 
going to make for dinner. And yet, you arrive at work. In fact, it can become so auto¬ 
matic, you drive to work without meaning to. A 


A The first time I used this analogy, I had come back to give a talk at the lab where I had done my 
postdoctoral work in Tucson, Arizona. I was staying with my former mentors, in their guest house. 
When I worked in their lab (for about three years), I lived on the east side of town, on the south side 
near the airbase. They also lived on the east side of town, but north, by the mountains. So the first part 
of driving from the lab to their house was the same as the first part of what had been my usual trip 
home for years. And, even fully conscious of this issue that I had just laid out in my talk that morning, 
I turned south when I got to the corner and didn’t realize my mistake until I had turned onto the street 
on which I used to five. 
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As we will see later in this section, these two processes (the conscious, attention- 
demanding, map-based,planningpro cess and the automatic, low-attention, sequence-based, 
routine process) actually reflect two different systems that process information about 
the decision in fundamentally different ways, and depend on different brain structures. 1 
Scientists studying decision-making have identified at least four separate action-selection 
processes: 

• A hardwired system that reacts quickly to immediately sensed direct dangers and 
events. These are your reflexes, genetically wired into your spinal cord, your periph¬ 
eral nervous system, and your central brainstem. 

• A system that predicts outcomes and reacts in a genetically prewired way to those 
outcomes. Computationally, what is learned in this system is a hypothesized causal 
relationship between the cues and the outcomes. The actions are not learned, they 
are “released” as appropriate responses to the expected outcome. Throughout this 
book, I will refer to this as the Pavlovian action-selection system. 

• A system that deliberates over decisions, requiring a lot of resources, but also 
capable of complex planning and flexibility. I will refer to this as the Deliberative 
system, but it has also been described as goal-directed learning, as model-based 
decision-making, as the action-outcome system, and as the locale or place naviga¬ 
tion system. 2 

• A system that simply learns the best action to take in a given situation. Once 
that association is stored, the decision process is very simple, but is inflexible. I 
will refer to this as the Procedural action-selection system as it is very involved 
in learned motor sequences (procedures), but it has also been referred to as 
the habit, stimulus-response, stimulus-action, cached-action, and taxon naviga¬ 
tion system. 3 

These four systems are generally sufficient to explain action-selection in the mam¬ 
malian brain; B however, they each have internal subtleties that we will address as we 
examine each system separately. In addition, their interactions produce subtleties not 
directly obvious from their individual computational components. 

For these systems to work in the real world, we need several additional support sys¬ 
tems. At this point, it is not clear whether these support systems are separate compo¬ 
nents that are shared among the decision-making systems, or whether there are separate 
versions of each, so that each decision-making system has its own copy of each support 
system. Alternatively, these support systems could be full-fledged systems in their own 
right, with their own internal components. Finally, it’s possible that the support sys¬ 
tems correspond in some way to some aspect of one or more of the decision-making 
systems. Nevertheless, breakdowns can occur in the support systems as well, 3 and we 


B Some scientists have suggested a fifth system based on recollection of a single event in the past 4 
(“It worked last time, maybe it will work again”). It is not clear, however, whether this is a different 
system or whether this is a special case of the Deliberative system we will see below and explore in 
depth in Chapter 9. The brain structures involved in this fifth system seem to be the same ones that are 
involved in the Deliberative system. 
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will need to include them in order to complete our description of the machinery of 
decision-making. 

• First, of course, one needs a motor control system that produces the muscle move¬ 
ments, the effectors that move the body and take the actual action. As the ques¬ 
tions being addressed in this book are primarily about the selection of the action, 
not the action itself, we will not spend a lot of time on the spinal motor control 
system, except to say that actions are generally stored as processes more complex 
than moving a single muscle. 

• Second, one needs perceptual systems that receive and interpret the basic sensory 
signals to determine what information those signals carry. These systems need to 
recognize objects and their basic properties ( What is it? Where is it?). We will dis¬ 
cuss these in terms of the detection of features and the integration of information. 

• In addition, however, one needs a system that recognizes which are the important 
cues on which to base one’s decision, whether in terms of external cues in the 
world or internal body-related cues. Although many scientific decision-making 
literatures refer to this process as “stimulus identification,” I think that term leads 
us to imagine that reactions are made to a single stimulus, while clearly we are rec¬ 
ognizing multiple stimuli and integrating them into a recognition of the situation 
we are in. Thus, I will refer to this as the situation-recognition system. 

• Finally, one needs a system that encodes one’s goals and desires. This system is 
going to depend on one’s internal (evolutionary) needs, but will also need to 
mediate conflicting goals, needs, and desires. For obvious reasons, I will refer to 
this as the motivational system. 

Although I will refer to these different aspects of decision-making as “systems,” I don’t 
want you to take them as different, separate modules. A good example of what I mean by 
the term “system” is the electric and gas engines in the new hybrid cars (like the Toyota 
Prius) —the car can be driven by the electric motor or by the gas motor, but both engines 
still share the same drive train and the same accelerator pedal. Some parts are shared, and 
some parts are separate. In the end, however, the goal of both engine systems is to make 
the car go. And the car still requires an additional steering system to guide it on the cor¬ 
rect course. Similarly, what matters for evolution is the behavior of the organism itself, 
not the separate decision-making systems. The decision-making systems are separate in 
that they require different information processing and involve overlapping, but also dif¬ 
ferent, neural structures, even though they all interact to produce behavior. We will find 
it useful to separate these systems as a first step in our analysis, but we will need to come 
back to their interaction in order to understand behavior (and its failure modes). 

Although we like to think of ourselves as a unitary decision-maker with a single set of 
desires, we all have experiences in which these multiple systems can come into conflict 
with each other. The reason that multiple systems evolved is that they each have advan¬ 
tages and disadvantages, and they are each better at some tasks and worse at others. If 
one can successfully select which system is going to be best at each time, one can gain the 
advantages of each and diminish the disadvantages of each. We will talk later about what 
is known about the mechanism by which we mediate between these systems, but there is 
still a lot of debate in the scientific community about exactly how that mediation happens. 
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The four action-selection systems 

Reflexes 

Our reflexes are the simplest decision-making systems we have. Scientists study¬ 
ing decision-making often dismiss the reflex as not being a decision-making sys¬ 
tem because it’s too simple. But by our definition of decision-making as selecting an 
action, we have to include a reflex as a decision. Because reflexes interact with the other 
decision-making systems, we will find that it is useful to include reflexes in our taxon¬ 
omy of decision-making systems. 

Let’s take an example reflex. If your hand touches something hot enough to burn it 
(say a candle flame), you pull your hand away quickly. This is something that an animal 
wants to do as quickly as possible. If we took the time to think, “That’s hot. Should I 
pull my hand away?” the flame would have done damage to our fingers. Similarly, we 
don’t have time to learn to pull our hand away from the fire; evolution wants that to 
happen correctly the first time. We can think of reflexes as simple operations (actions 
to be selected in reaction to specific conditions) that have been learned over evolu¬ 
tionary time, but that once learned within a species, are hardwired within a given 
individual. These twin issues of the time it takes to compute an answer and the time it 
takes to learn the right answer are critical advantages and disadvantages of each of the 
action-selection systems. 

Proof that a reflex really is a decision can be seen in its interaction with the other 
action-selection systems. In David Lean’s Lawrence of Arabia, there is a famous scene 
where T. E. Lawrence, played by Peter O’Toole, waits for the match to burn all the way 
down to his fingers. His companion then tries it and complains that it hurts. “Of course,” 
replies Lawrence, “but the trick is not minding that it hurts.” It is possible to override the 
default Reflex action-selection with a different, more conscious system. This interaction 
between multiple systems as an important part of how we make decisions is something 
we will return to several times in this book. 


The Pavlovian action-selection system 

Pavlovian action-selection is also computationally simple in that it can only produce 
actions that have been learned over an evolutionary timescale (often called unconditioned 
responses), 6 but it differs from the Reflex action-selection system in that it learns. c 


c I have struggled with finding the right terminology to use for this first learning-capable system. 
The computational mechanisms underlying the first system are very clear, but we need a term for it. 
(I was tempted at one point to call it “System X” to avoid all of the historical baggage associated with 
the term “Pavlovian”) Historically, learning in animals (including humans) has been assumed to arise 
from two different systems, one when no action was required to get the reward (Pavlovian) and a 
different one when action was required (Instrumental). 7 

In Pavlov s original experiments, the sound occurred (the “bell”) and then the food appeared. The 
dog didn't have to do anything to get the food, so that was identified as the first system, which was 
called “Pavlovian.” If, on the other hand, the dog had to push a lever with its paw after the bell rang to 
get food, that would be “instrumental.” 


( continued ) 
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The Pavlovian system learns stimuli that predict outcomes, but it does not learn the 
actions to take with those outcomes. 14 Pavlovian learning is named after Ivan Pavlov 
(1849-1936), the great Russian neuroscientist. Supposedly, Pavlov rang a bell before 
feeding a dog and found that the dog learned to salivate to the bell in anticipation of 
the food. D 

When describing Pavlovian learning, scientists often describe the stimuli as 
“releasing specific actions.” 19 In classic animal learning theory, there is an uncondi¬ 
tioned stimulus (the food being presented to Pavlov s dog), an unconditioned response 
(salivating), and a conditioning stimulus (the bell). In Pavlovian learning, the uncon¬ 
ditioned response shifts in time from the unconditioned stimulus to the condition¬ 
ing stimulus. Interpretations of Pavlovian learning tend to ignore the action-selection 
component of the learning, suggesting that the unconditioned response is simply a 
tool to identify an association between the conditioning stimulus and the uncondi¬ 
tioned stimulus. 20 

These interpretations are based in part because the same training that makes 
animals take the unconditioned response after seeing the training cue (Pavlovian 
action-selection) also often makes animals more likely to take other, untrained 
actions that they know will reach the unconditioned stimulus (as if they had an 
increased motivation to reach that unconditioned stimulus). 21 This is a process called 
Pavlovian-to-instrumental transfer, which we will examine in depth when we reach 
our discussion of motivation, below. Recent data suggest that the two effects (release 


We now know that these systems are more complicated—for example, the traditional term “instrumen¬ 
tal” includes both the Deliberative and Procedural systems, which select actions using very different com¬ 
putational processes, 8 and Pavlovian learning is really about action-selection appropriate to cue-outcome 
expectations. 9 In addition, these two learning systems interact in a process called “Pavlovian-to-instrumental 
transfer,” 10 which we will identify with the motivational support system (later). 

As we will see in our in-depth discussion of Pavlovian systems (Chapter 8), this “Pavlovian” system 
is related to basic survival instincts and to emotions. 11 However, neither of those terms captures all 
of the computational issues either, and they both have their own historical baggage associated with 
them—animal behavior is about genetic success, not survival, 12 and the direct relationship between 
what we are calling the Pavlovian action-selection system and emotion is controversial. 13 Therefore, I 
will use the term “Pavlovian action-selection” throughout this book as a place holder for a better term 
for this first learning-based action-selection system. 

D Pavlov was the first winner of the Nobel prize in Medicine in 1904, but surprisingly to many 
people today, what he won it for was not the learning studies that we know him for today; rather, he 
won it for his work on digestion and salivation. 15 After his Nobel prize, he felt famous enough to risk 
working on learning, starting a tradition of Nobel prizewinners turning from unrelated medical topics 
to psychology and neuroscience. Other examples include Francis Crick 16 (1916-2004, Nobel prize 
in 1962 for DNA) and Gerry Edelman 17 (1929-, Nobel prize in 1972 for antibodies in the immune 
system), both of whom turned to work on consciousness after winning their prizes. 

As with most popular descriptions of the process of science, the story taught in introductory sci¬ 
ence classes that Pavlov had an explicit hypothesis that he then went to test belies the actual process 
of discovery in science. In fact, Pavlov was working on the mechanisms of digestion. 18 As part of his 
animal research setup, he had to put the dog into a harness that was counterbalanced by a heavyweight. 
When the weight was dropped, Pavlov noticed that the dog started to salivate. Pavlov recognized that 
something interesting was happening and decided to chase it. 
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of unconditioned responses to conditioned stimuli [Pavlovian action-selection] 
and an increase in motivation as evidenced by increased actions taken by other 
systems [Pavlovian-to-instrumental transfer]) are dissociable. They are differently 
affected by different brain structure lesions and they are differently affected by dif¬ 
ferent pharmacological manipulations. 22 The relationship between the Pavlovian 
action-selection and motivational systems is still an open question being actively 
pursued by scientists. 

However, we must not forget that the animal is, in fact, taking an action. Recent inter¬ 
pretations suggest that Pavlovian learning plays a role in an action-selection system, in 
which the unconditioned responses are actions learned over an evolutionary timescale. 23 
Imagine living on the savannah where there are lions that might hunt you. You can learn 
that the rustle in the brush predicts that a lion is stalking you and you can learn to run 
from the rustling grass, but you don’t get a chance to learn to run from the lion—you’ve 
got to get that right the first time. The Pavlovian action-selection system can learn that 
the rustle in the grass is a lion stalking you, which leads to fear, and running, but it can’t 
learn that the right response is to do jumping jacks. 

A great way to understand the Pavlovian system is the phenomenon of sign-trackers 
and goal-trackers . 24 Imagine rats being placed in an environment with a food-release 
port and a light, separated by some small but significant distance. The light comes on, 
and a few moments later food is delivered at the food port. The connection between 
the light and the food is independent of what the rat does—the light simply predicts 
that the food will be available. The right thing for the rat to do is to go to the food 
port when the light comes on. (This would be “goal-tracking.”) Some rats, however, 
go to the light during the intervening interval and gnaw on it. They still go to the food 
when the food is delivered, but they have wasted time and energy going out of their 
way to the light first. (This would be “sign-tracking.”) In fact, if a sign-tracking rat has 
had enough experience with the light-food association, and you stop giving the food 
if the rat goes to the light and give it only if the rat goes directly to the food port, the 
rat has a devil of a time stopping going to the light. 25 The Pavlovian association is too 
strong. 

Current thinking suggests that Pavlovian action-selection is related to what we recog¬ 
nize as emotional responses, 26 although the specific relationship between the Pavlovian 
action and the linguistically labeled emotion is controversial. 27 Some authors have sug¬ 
gested that emotional responses are Pavlovian actions changing internal responses (heart 
rate, salivation, etc.). 28 Others have suggested that the things we label emotions are a 
categorization process applied to these internal responses. 29 A discussion of categoriza¬ 
tion processes in mammalian brains (including humans) can be found in Appendix C. 
We will comeback to the relationships between Pavlovian action-selection and emotion 
in Chapter 8. 

Although flexible in the stimuli it can react to, the set of actions available to the 
Pavlovian action-selection system is greatly limited. Pavlovian responses can only 
transfer a simple “unconditioned response” to an earlier stimulus. This makes it more 
flexible than simple reflexes, but the Pavlovian action-selection system cannot take an 
arbitrary action in response to an arbitrary stimulus. For that, we need a more complex 
decision-making machinery, like the Deliberative or Procedural systems. 
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The Deliberative system is more flexible. It can take any action in any situation, but it 
is a computationally expensive system. In humans, deliberation entails planning and 
consists of a conscious investigation of the future. This is particularly useful for large 
one-time decisions, where one cannot try the individual options multiple times to 
determine the value of each. 

For example, a friend of mine spent the last month deciding between two universi¬ 
ties who had made him job offers. Each offer had advantages and disadvantages. This 
was not a decision that he could make ten times, observe what the answers are each 
time, and then slowly learn to make the correct choice. Obviously, it was not a deci¬ 
sion with an action that evolved over generations.® Instead, he had to imagine himself 
in those two futures— What would life be like at University One? What would life be like at 
University Two ? Through his life experience and through discussion with his colleagues,® 
he tried to work out the consequences of his decision. From that complex deliberation, 
he worked out what he thought was the best choice. 

This is a process called episodic future thinking. 30 We will return to the issue of epi¬ 
sodic future thinking in later chapters, but the concept is that when trying to plan a 
complex future event, one needs to draw together memories and experiences from lots 
of sources to imagine what the potential future might be like. By imagining ourselves 
in that future event, we can determine whether it is something we want to happen or 
something we want to avoid. As we will see, both the hippocampus and the prefrontal 
cortex are critically involved in episodic future thinking. 31iG 

A critical question that has plagued psychology for years is Do animals deliberate as 
well? 34 We can’t ask a rat what it is thinking about, but we can decode information from 
neural signals. If we know what information is represented by an ensemble of cells, then 
we can decode that information from the activity in those cells. (See Appendix B for 
a discussion of current technology and methods that enable the decoding of neural 


E I think it is pretty unlikely that “take the job at University One” can be encoded in one’s genes. 
Clearly this question can’t be solved by reflexes. Similarly, although this is certainly not a decision that 
the Pavlovian action-selection system can make (I can’t imagine “sign on the dotted line” to be a geneti¬ 
cally hard-coded response to any outcome), one could argue that the Pavlovian system influenced his 
decision, because he felt drawn to one place or another. I suspect that there is Pavlovian involvement 
when someone says, “I want to go back to that university because it’s in my hometown where I grew up 
and I want to go home” (or Til never go back to that university because it’s in my hometown where I 
grew up,” which I’ve also heard!). In any case, the ultimate decision to go to a new job or not is one that 
almost certainly engages the Deliberative decision-making system. 

F This is one of the advantages of human language—we can improve our prediction from observa¬ 
tions and discussions with others. 

G The hippocampus is often discussed as being critical to episodic memory, meaning memories of 
past events in one’s life. 32 The role of the hippocampus in imagining the future (episodic future think¬ 
ing) suggests a particularly interesting hypothesis for the role of the hippocampus in memories of the 
past: perhaps we don’t remember the past—perhaps we actually imagine our pasts. Perhaps we are 
using the system evolved to imagine the future when we try to remember these past events. This may be 
an explanation for why our memories for past events are so fragile and so easily manipulable. 33 
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signals.) The hippocampus is a critical part of episodic future thinking in humans. 35 
In rats, the information represented in hippocampal cells is first the location of the ani¬ 
mal in an environment (these are known as place cells). 36 From the activity in these cells, 
we can decode the location represented at a moment in time by the cells. 37 So, if a rat 
were to imagine being at another place in the environment, we should be able to decode 
that imagination by looking at the set of active hippocampal cells. 38 

Rats certainly look like they deliberate over choices. When rats come to a difficult 
or recently changed choice (say at a T-intersection in a maze), they sometimes pause, 
turning first one way and then the other. 39 When this behavior was first observed by 
Karl Muenzinger and his student Evelyn Gentry in 1931, they called it “vicarious trial 
and error” because they thought that the rat was vicariously trying out the possibilities. 
Just as my friend imagined himself at his two future jobs to try to decide between them, 
the rat was imagining itself taking the two choices. These ideas were further developed 
by Edward Tolman in the 1930s and 1940s, who showed that these vicarious trial and 
error processes occurred when we ourselves would deliberate over choices and argued 
explicitly that the animals were representing the future consciously. 11 

Unfortunately, Tolman did not have a mathematical way to explain his ideas and was 
attacked because he “neglected to predict what the rat will do.” The psychologist Edwin 
Guthrie, in a 1937 review article, complained that “So far as the theory is concerned the rat 
is left buried in thought.” 42 Tolman, however, was writing before the invention of the mod¬ 
em computer and had no language to say “yes, the rat is buried in thought because it is 
doing a complex calculation.” In our modem world, we have all experienced the idea of 
computational complexity—algorithms take time to process. When our computers go 
searching for a file in a directory. When a website takes time to load. When a computer 
opponent in a game takes time before responding. It is not so strange to us for a com¬ 
puter to be “lost in thought.” And, in fact, Tolman (following Muenzinger and Gentry) had 
directly observed rats pausing at those choice points, presumably “lost in thought.” 

In addition, of course, Tolman and his colleagues did not have the neurophysiologi¬ 
cal tools we have now. Just as humans with hippocampal damage cannot imagine future 
events, 43 so too rats with hippocampal damage no longer pause at choice points, nor 
do they show as much vicarious trial and error. 44 We can now decode representations 
of place from recorded ensembles of hippocampal neurons (Appendix B). When we 
did this in my laboratory, we found that during those paused “vicarious trial and error” 
events, the hippocampal representation swept ahead of the animal, first down one 
potential future choice, then down the other. 45 Just as my friend was imagining what it 
would be like to take one job or the other—so, too, the rat was imagining what would 
happen if it went running down the leftward path or down the rightward path. The most 


H Whether or not animals are conscious is, of course, a big question that hinges in part on the ques¬ 
tion What is consciousness^ When I used to give talks on our vicarious trial and error data, I used to like 
to answer the inevitable question about whether the rats were consciously deliberating over the future 
by saying that I thought it was the same process as humans, but that “neither Tolman nor I would ever 
use the word ‘consciousness’.” One day, after hearing me give one of these talks, Adam Johnson handed 
me Tolman’s 1932 textbook and showed me the section titled “Vicarious trial and error is proof of con¬ 
sciousness in the rat.” 41 For now, let us simply say that the question of consciousness is a difficult one 
we will return to at the end of the book (Chapter 24). 
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important observation we made was that these representations were sequential, coher¬ 
ent, and serial. They were sequential in that they consisted of a sequence of cells firing 
in the correct order down a path. They were coherent in that the sequences consisted 
of accurate representations of places ahead of the animal. And they were serial in that 
they went down one path and then the other, not both. Our rats really were imagining 
the future. 1 

It’s one thing to imagine the future. But how did our rats escape being lost in thought? 
How did our rats make the actual decision? The next step in deliberation is evaluation. 
Our rat needs to go beyond saying “going left will get me banana-flavored food pel¬ 
lets” to say “and banana-flavored food pellets are good.” Two structures involved in 
evaluation are the ventral striatum and the orbitofrontal cortex. Both of these struc¬ 
tures are involved in motivation and are often dysfunctional in drug addiction and 
other motivation-problem syndromes. 47 Moreover, the ventral striatum receives direct 
input from the hippocampal formation. 48 Orbitofrontal cortex firing is modulated by 
hippocampal integrity. 49 What were these structures doing when the hippocampus 
represented these potential futures? 

Just as we can record neural ensembles from the hippocampus and decode spatial 
representation, we can record neural ensembles from the ventral striatum or the orb¬ 
itofrontal cortex and decode reward-related representations. We found that both ven¬ 
tral striatal and orbitofrontal reward-related cells (cells that normally respond when the 
animal gets reward) showed extra activity at the choice point where vicarious trial and 
error was occurring. 50 Muenzinger, Gentry, and Tolman were entirely correct—when 
these rats pause and look back and forth at choice points, they are vicariously trying out 
the potential choices, searching through the future, and evaluating those possibilities. 

The deliberative decision-making system has a lot of advantages. It’s extremely flexi¬ 
ble. Knowing that going left can get you banana-flavored food pellets does not force you 
to go left. You can make a decision about which college to go to, which job to take, which 
city to move to, without having to spend years trying them. Knowing that one company 
has made you a job offer does not commit you to taking that offer. But deliberation 
is computationally expensive. It takes resources and time to calculate those potential 
future possibilities. If you are faced with the same situation every day and the appropri¬ 
ate action is the same every time, then there’s no reason to waste that time re-planning 
and re-evaluating those options—just learn what the right choice is and take it. 


Procedural action-selection 

The Procedural system is, on the surface, much simpler—it simply needs to recognize 
the situation and then select a stored action or sequence of actions. 51 This is the classic 
stimulus-response theory of action-selection. The Procedural action-selection system 
differs from the three other systems. Unlike the hardwired reflexes, the Procedural sys¬ 
tem can learn to take any action. Like the Pavlovian system, it can learn to work from 


1 It is important to note that our rats were only representing the immediate future. If this is episodic 
future thinking, 46 it is future thinking of an immediate future only a few seconds ahead. Humans can 
imagine futures days, months, years, even centuries away. 
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any given situation, but unlike the Pavlovian system, it can learn to take any action in 
response to that recognized situation. In contrast to the Deliberative system, Procedural 
action-selection can work very quickly, but it doesn’t include a representation of the 
outcome, nor is it flexible enough to change easily. In short, Procedural learning associ¬ 
ates actions with situations. It can take the action quickly because it is just looking up 
which action to take, but because it is just looking things up, it doesn’t have the flexibil¬ 
ity to easily change if needed. 

To see the distinction between the Deliberative and Procedural systems, we can 
imagine a rat trained to push a lever to receive fruit-flavored food pellets. Once the rat 
has learned the task (push the lever, get fruit-flavored food), when the rat is placed back 
in the experimental box, it will push the lever and eat the food. We can imagine two 
possible learning mechanisms here: 52 "If I push the lever, fruit-flavored food comes 
out. I like fruit-flavored food. I’ll push the lever” or “Pushing the lever is a good thing 
to do. When the lever appears, I’ll push the lever.” The first logic corresponds to the 
Deliberative (search-based, predicting outcomes from actions) system, while the sec¬ 
ond logic corresponds to the Procedural (cached-action, automatic, habit, stimulus- 
response) system. 

These two cognitive processes can be differentiated by changing the value of the 
fruit-flavored food to the rat. For example, the rat can be given fruit-flavored food pel¬ 
lets and then given lithium chloride, a chemical that does not hurt the rat but makes it 
feel ill. 53 (Lithium chloride often makes humans vomit.) When the rat is next placed 
into the experimental box, if it is using the Deliberative system, the logic becomes “If I 
push the lever, fruit-flavored food comes out. Yuck! That stuff is disgusting. Don’t push 
the lever.” If, instead, the rat is using the Procedural system, then the logic becomes 
“There’s that lever. I should push it.” When fruit-flavored food comes out, the rat ignores 
it and doesn’t eat it—but it still pushes the lever. These two reactions depend on dif¬ 
ferent brain structures, occur after different training paradigms, and require different 
computational calculations. 54 

Computationally, the Procedural system is very simple: learn the right action or 
sequence of actions, associate it with a recognized situation. 55 (Actually, most models 
associate a value with each available action in each situation, storing values of situation- 
action pairs, and then take the action with the highest value. 56 ) Then the next time you 
face that situation, you know exactly what to do. This is very much the concept of a non- 
conscious habit that we are all familiar with. When a sports star learns to react quickly 
“without thinking,” it’s the Procedural system that’s learning. When soldiers or police 
learn to respond lightning-quick to danger, it’s their Procedural systems that are learning 
those responses. 

The old-school psychology literature often talked about this system as “stimulus- 
response,” with the idea that the specific responses (press a lever) are becoming attached 
to specific stimuli (the light turns on), but we know now that the responses can be pretty 
complex (throw the ball to the receiver) and that the stimuli are better described as full 
situations (he’s open). Extensive experimental work on the mechanisms of motor control 
shows that the Procedural learning process can recognize very complex situations and 
can learn very complex action sequences. 57 Part of the key to being a successful sports 
star or making the correct police response is the ability to recognize the situation and 
categorize it correctly. 58 Once the situation has been categorized, the action-selection 
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process is simple. One way to think of the difference between the Deliberative and 
Procedural systems is that the Procedural system shifts all of the decision-making 
complexity into the situation-recognition component. 

One of my favorite scenes of this is in the movie Men in Black , 59 where the new 
recruit (Agent J-to-be, played by Will Smith) is faced with a simulated city scene and 
decides to ignore all the cutouts of scary aliens and shoot the little girl. When asked 
why he had shot “little Tiffany” by Chief Agent Zed, he responds by explaining how 
each alien looked scary but was just “trying to get home” or was “actually sneezing,” 
but “little Tiffany was an eight-year-old girl carrying quantum mechanics textbooks 
through an unsafe city street at night.” The point, of course, is that Agent J had excel¬ 
lent situation-categorization skills and noticed the subtleties (such as the scary alien’s 
handkerchief and little Tiffany’s quantum textbooks) and could react quickly to the 
correct situation. 

Although describing Procedural action-selection as storing an action seems simple 
enough—recognize the situation and take the stored action—there are two important 
complexities in how this system works. First, one has to learn what the right action is. 
Unlike the Deliberative system, the Procedural system is very inflexible; which action is 
stored as the best choice cannot be updated on the fly. We will see later that overriding 
a stored action is difficult and requires additional neural circuitry (self-control systems 
in the prefrontal cortex, Chapter 15). Second, the circuitry in the mammalian brain 
(including in the human) seems to include separate “go” and “don’t go” circuits—one 
system learns “in this situation, take this action,” while another system learns “in that 
situation, don’t.” 60 

The current data strongly suggest that the Procedural system includes the basal 
ganglia, particularly the dorsal and lateral basal ganglia circuits, starting with the cau¬ 
date nucleus and the putamen, which receive inputs from the cortex. 61 These synaptic 
connections between the cortical circuits (which are believed to represent information 
about the world) are trained up by dopamine signals. 62 Ifyou remember our discussion of 
value, euphoria, and the do-it-again signal (Chapter 4), you’ll remember that dopamine 
signaled the error in our prediction of how valuable things were 63 —if things came out 
better than we expected, there was an increase in dopamine and we wanted to increase 
our willingness to take an action; if things were worse than we expected, there was a 
decrease in dopamine and we wanted to decrease our willingness to take an action; and 
if things came out as expected, then we didn’t need to learn anything about what to do 
in that situation. The connection between the cortex and the caudate/putamen encodes 
our “willingness to take an action” and is trained up by these dopamine signals. 


Support systems 

In addition to the four action-selection systems, we need four additional components, 
a physical action system, which can physically take the action that’s been selected, a 
perceptual system, which can recognize the objects in the world around us, a situ¬ 
ation recognition system that categorizes stimuli into “situations,” and a motiva¬ 
tional system that identifies what we need next and how desperate we are to satisfy 
those needs. 
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Taking a physical action 

Most actions require that we move our muscles. This is true whether we are running 
down a street, turning to face an adversary, smiling or frowning, or signing on a dotted 
line. In the end, we move our muscles and have an effect on the world. Even speech is 
muscle movement. As a support system, this would often be referred to as the “motor 
control” system, because it is the final step in physically interacting with the world. For 
completeness, it is probably worth including nonmotor actions within this support sys¬ 
tem. For example, in our description of Pavlovian action-selection, Pavlovs dogs were 
salivating, releasing a liquid into their mouths, which is not really muscle movement. 
Similarly, getting overheated leads to sweating, which again is not really muscle 
movement. Nevertheless, most of the effects of the physical action system are motor 
movements. 

Details of how muscles work can be found in any medical textbook. Many neurosci¬ 
ence textbooks have good descriptions. 64 The important issue for us is that within the 
motor control (physical action) system, the building blocks are not individual muscle 
movements, but rather components made of sets of related movements, called muscle 
synergies , 65 But even within those muscle synergies, we do not place one foot down, 
heel to toe, place the other down, and lift the first foot; we do not put our feet one 
step at a time; instead, we walk? Human walking, like a fish or lamprey swimming, is 
an example of a process called a central pattern generator, so called because it is able 
to generate a sequence centrally, without external input. 67 Central pattern generators 
can continue to oscillate without input, but they can be modified directly from external 
sensory cues, and also from top-down signals (presumably from the action-selection 
systems themselves). 

Since our target in this book is to understand the action-selection process, we are 
going to sidestep the question of what those actions are at the individual motor control 
level, but obviously these processes can be very important clinically. 


Perception 

Much like the thermostat that we started with at the beginning of the book (Chapter 2), 
to take the appropriate actions in the world, one needs to perceive the world. Of course, 
our perceptual systems are much more complex than simply measuring the tempera¬ 
ture. Human perceptual systems take information about the world from sensory sig¬ 
nals, process that information, and interpret it. Although our eyes transform light 
(photons) into neural activity, we don’t see the photons themselves, we see objects— 
the tree outside my window, the chickadee hopping from branch to branch. Similarly, 
our ears detect vibrations in the air, but we hear sounds—the whistle of the wind, the 
song of the chickadee. 

As with the physical action system, details of how perceptual systems work can be 
found in many neuroscience textbooks. Neil Carlson’s Physiology of Behavior and Avi 


J It is, of course, possible to consciously place our feet one step at a time, but this entails one of our 
action-selection systems taking control of a subset of motor movements. This is an example of subsump¬ 
tion architecture, in which new systems do not replace old ones, but add control loops to them. 66 
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Chaudhuri’s Fundamentals of Sensory Perception are excellent starting points. We are not 
going to spend much time on the transition from physical reality to neural activity (pho¬ 
tons to sight; vibrations to sound), but we will take some time to explore how the neu¬ 
ral systems categorize perception to identify objects and pull information out of those 
sensory signals (see Chapter ll). 


Recognizing the situation 

When psychologists classically tested animals in learning experiments, they used individ¬ 
ually identifiable stimuli so that they could control the variables in their experiments. 68;K 
But what’s the important cue in the world right now? I am writing this paragraph on 
a black Lenovo laptop, sitting at a desk covered in a dark-brown woodgrain and glass, 
listening to Carlos Santana’s wailing guitar, looking out the window at the darkness (it’s 
6 a.m. in winter). Outside, a streetlight illuminates the empty intersection, the shadow 
of a tree looms over my window, there’s snow on the ground, and I can see the snow¬ 
flakes falling. The light over my desk reflects off the window. Next to me are my notes. 
Which of these cues are important to the situation? If something were to happen, what 
would I identify as important? 

Although it’s controversial, my current hunch is that the neocortex is one big 
situation-recognition machine. Basically, each cortical system is categorizing informa¬ 
tion about the world. The visual cortex recognizes features about sights you see; the 
auditory cortex recognizes features about sounds you hear. There are also components 
of the cortex that recognize more abstract features—the parietal cortex recognizes the 
location of objects around you; the temporal cortex recognizes what those objects are. 
There is an area of the cortex that recognizes faces, and an area that recognizes the room 
you are in. 70 Even the motor cortex can be seen as a categorization of potential actions. 71 
As one progresses from the back of the brain (where the sensory cortices tend to be) to 
the front of the brain, the categories get more and more abstract, but the cortical struc¬ 
ture is remarkably conserved and the structure of the categorization process does not 
seem to change much. 72 

Models aimed at each of these individual cortices all seem to work through a process 
called content-addressable memory. 73 Content-addressable memory can be distinguished 
from the kind of indexed memory that we are familiar with in typical computers because 
content-addressable memory recalls the full memory from part of the memory, while 
indexed memory is recalled from an unrelated identifier. An image of your daughter’s 
birthday party stored on your computer might be titled IMG342 5 . jpg (indexed 
memory, no relation between title and subject), but the memory would be recalled in 
your brain from a smile on her face, cake, candles getting blown out... 


K Of course, the experiments take place in the presence of other abundant cues, such as the experi¬ 
mental box, room, or maze. These other cues are usually identified in a catchall description called the 
“context.” The specific differences between context and cues are very complex and subtle and seem 
to have to do with differences between environmental cues (the room you’re in) and small object 
cues (a light or tone) and with differences in stability (contexts are stable) and variability (object cues 
change quickly). Nevertheless, it appears clear that context and individual cues are processed by dif¬ 
ferent brain systems. 69 
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Content-addressable memory is a process in which partial patterns become com¬ 
pleted. The computational processes that underlie content-addressable memories are 
well understood and can be thought of as a means of categorization. A thorough descrip¬ 
tion of those computational processes, their neural implementation, and how they lead 
to categorization can be found in Appendix C. 

The hypothesis that the neocortex is machinery for situation-categorization pro¬ 
vides a potential explanation for both the extensive increase in neocortical surface area 
in humans and our amazing abilities that seem to be uniquely human—we are better at 
categorizing situations, better at learning the causal structure of the world (what leads 
to what), and better at abstracting this information. A more complex categorization of 
the world would lead to an improved ability to generalize from one situation to another 
and an improved ability to modify what worked in one situation for use in another. 
However, this also provides a potential explanation for those cognitive diseases and dys¬ 
functions that are uniquely human, such as the paranoia and incorrect causal structure 
seen in schizophrenic delusions 74 —our ability to recognize the complexity of the world 
can lead us to find conspiracies and interactions that might not actually be present. 

In a sense, the situation-recognition system is learning the structure of the world. It 
learns what remains constant in a given situation and what changes. It learns what leads 
to what. This has sometimes been referred to as semantic knowledge, 75 the facts about 
our world, our place in it, the stories we tell ourselves. Semantic knowledge in humans 
includes both the categorization of the features of the world ( that's a chair, hut that’s a 
table), facts about the world (this table is made of wood), and the narrative structure we 
live in (I have to go home so that my family can eat dinner together tonight). This semantic 
knowledge forms the inputs to our action-selection systems. Changing that knowledge 
changes the causal structure that leads to expectations (a rustle in the brush predicts a 
lion, Pavlovian action-selection), the expected consequences of our actions ( too much 
alcohol will get you drunk, Deliberative), and the set of situations in which we react ( stop 
at a red light, Procedural). We’ll explore this system and how changing it changes our 
decisions in depth in Chapter 12. 


Motivation 

Finally, we need a motivational system, which includes two important components: 
first, What are the goals we need to achieve? (we want to take different actions if were 
hungry or if we’re thirsty) and second, How hard should we be working for those goals? 
(how thirsty we are will translate into how desperate we are to find water). We will see 
that both of these issues are related to the concept of value that we first looked at in 
Chapter 3. 

Historically, these issues have been addressed through the reduction of innate 
“drives.” 76 But what creates those drives? Some motivations certainly exist to maintain 
our internal homeostatic balance, L but people often eat when they are not hungry. Why? 


L Homeostasis is the maintenance of an internal parameter at a given level. For example, mammals 
are warm-blooded and try to maintain their temperature at a given level. When cold, we shiver, wrap 
ourselves up in warm clothes, or seek heat sources. When hot, we sweat, pant, or seek pools to swim in. 
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To understand this, two additional concepts need to be taken into account—first, that of 
intrinsic reward functions 77 and second, that of the role of learning in value calculations. 78 

Our complex decision-making system evolved because animals who made better 
decisions were more able to survive and to find better mates, were more likely to pro¬ 
create, and were more likely to have offspring that survived themselves. 79 But there is 
nothing in this process that requires that the proximal reasons for decisions be related to 
any of these evolutionary goals. Instead, what evolution tends to do is find goals that are 
correlated with these factors. For example, an animal that lived in an environment where 
food was scarce would be less likely to starve and more likely to survive if it ate whenever 
food was available, whether it was hungry or not. 80 But what happens when these intrin¬ 
sic processes no longer track survival? What would happen to a species that had evolved 
to eat whenever food was available but was now faced with an overabundance of food? 
It would get fat, and new diseases, such as diabetes, would appear. 81 

We can see examples of how intrinsic reward functions correlated to evolutionary 
success can get corrupted in many animals and many situations. Dogs love to chase large 
game (e.g., deer, moose, elk). They do this because that was how they got their food— 
that was their evolutionary niche. Domestic dogs chase cars. It is pretty clear that if a 
dog ever caught a car, it would not be able to eat it. But they chase cars for the sake of the 
chase. In humans, the classic example is pornography. Men like looking at pretty women 
because being attracted to a beautiful woman helped them judge who would be the best 
mates, M but reading Playboy or watching Internet porn is not going to help men find 
better mates. In fact, one could easily imagine an argument that these nonreproductive 
options impair the ability to find better mates. 

So, yes, we have drives that are directly related to evolutionary success (hunger, 
thirst, sex), and we have intrinsic reward functions that are correlated to evolutionary 
success (chasing prey, an attraction to beauty). But there is also a learned component 
to motivation in mammals. This learned component can be seen both in generalized 
increases (or decreases) in arousal and in specific increases or decreases in the willing¬ 
ness to work for a given reward. 

Does this learning component also occur in humans? Actually, this is one of the 
best-established effects in drug addicts: seeing the paraphernalia for their drug of choice 
leads to craving for that drug and an increase in actions leading to taking that drug. 85 
Marketing symbols (Coke, McDonalds) increase our internal perceptions of thirst or 
hunger and make people more likely to go get a soda or a hamburger. 86 These are all 
examples of the motivational system driving other action systems. 


M There’s now pretty good evidence that many (but not all) standards of beauty actually correlate 
with health and fertility (for both men and women). Of course, this is just another example of an intrin¬ 
sic reward function—certain features (such as a specific breast-to-waist-to-hip ratio, symmetry of the 
face) are correlated with fertility, so people are attracted to those features. 82 These features signal health 
and fertility and we call people with those features “beautiful.” Of course, in humans, having successful 
children who succeed at having their own children is much more complex than simple health, 83 which 
may be why humans find other humans with intelligence and talent (such as musicians) particularly 
attractive. 84 
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Motivation is a complex phenomenon related to the complexity of evaluation. It 
depends on intrinsic functions that we have evolved as animals. 87iN Motivation can also 
be modified through an interaction between the emotional, Pavlovian systems and the 
other, more complex decision-making systems (Deliberative, Procedural). 


Summary 

Behavioral decision-making occurs as a consequence of a number of different, inter¬ 
acting systems. We have identified four action-selection systems (Reflex, Pavlovian 
action-selection, Deliberative, Procedural) and four support systems (taking the physi¬ 
cal action, perception, situation-recognition, motivation). One of the remarkable things 
that has occurred over the past several decades is the convergence of different fields 
on these multiple decision-making systems. A number of fields have been examining 
how different agents interact with each other and with the world (e.g., human psychol¬ 
ogy and psychiatry, animal learning theory, robotics and control theory, artificial intel¬ 
ligence, and neuroscience, including both computational modeling and the new fields 
of neuroeconomics and computational psychiatry). 91 

One of the things that motivated me to write this book has been the observa¬ 
tion that these different fields seem to be converging on a similar categorization of 
decision-making systems. Even the debates within the fields are similar—for example, Is 
the value calculation within the search-based (Deliberative) and cached-action (Procedural) 
systems separate or unitary? 91 (I have argued here and elsewhere that they are separate, 
but the issue is far from settled.) 

For example, roboticists have come to the conclusion that one needs multiple levels 
of decision-making systems—fast, low-level, hardwired systems (don’t bump into walls; 
a Reflex system), as well as slower, search processes (slow but flexible; a Deliberative 
system), and fast, cached-action processes (fast but inflexible; a Procedural system). 93 

Computer scientists studying artificial intelligence and computational neuroscien¬ 
tists trying to model animal decision-making have identified these systems in terms of 
their expectations— Pavlovian systems create an expectation of an outcome and lead 
to unconditioned actions related to that outcome; Deliberative systems search through 
potential futures, produce an expectation of the outcome, and evaluate it; Procedural 
systems directly evaluate actions in response to specific situations. 94 

Similarly, learning theorists working in the fields of psychology and animal-learning 
identify three systems. 95 First, there is a stimulus-outcome system in which uncon¬ 
ditioned reactions associated with outcomes get transferred from being done in 
response to the outcome to being done in response to the stimulus. (Think Pavlov’s 
dogs salivating in response to a cue that predicts food reward.) Second, they identify 
an action-outcome system in which animals learn what outcomes will occur if they take 
a given action. These actions are taken to achieve that expected outcome. Third, they 


N One of the things we will see at the very end of this hook (Chapters 22 to 24) is evidence that part 
of human morality arises because we are social creatures who have evolved to live in large societies, 88 
with intrinsic motivation functions based on enforcing cooperation. 89 As Aristotle said, “we are political 
animals.” 90 
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identify a stimulus-response or stimulus-action system in which actions are taken in 
response to a stimulus. The second and third systems can be differentiated by revalua¬ 
tion and devaluation experiments, in which the value of the outcome is changed online. 
(Remember our example of the rat deciding whether or not to push the lever to get food 
that it didn’t like anymore.) 

So where does that leave us? We have identified four action-selection systems and 
four support systems. Although these systems are separable in that different brain struc¬ 
tures are involved in each of them, they also must interact in order to produce behav¬ 
ior. (The analogy brought up earlier was that of the electrical and gasoline systems in 
a hybrid car.) Each of these systems is a complex calculation in its own right and will 
require its own chapter for full treatment. In the rest of this part of the book, we will 
explore each of these components, the calculations they perform, and their neural sub¬ 
strates, and we will begin to notice potential failure points within them. Let s start with 
the simplest action-selection system, the reflex. 
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Reflexes 


The geography of the shell 
remembers swimming, 

the slow expansion and contraction of centuries, 
the memory of a distant shore, 
now only open sea. 


A reflex is a decision-making system because it includes taking an 
action. A reflex entails a simple response to a stimulus based on a 
rule learned over long, evolutionary timescales. Although reflexes 
can change within a lifespan, those changes are either predeter¬ 
mined genetically (as animals change from children to adults) or 
entail only very simple learning processes (limited primarily to 
habituation and adaptation effects). 


The reflex is the simplest decision-making system. It entails a simple rule-based reaction. 
In a sense, the thermostat is a reflex. When it is too hot, the thermostat turns on the air 
conditioning; when it is too cold, the thermostat turns on the heat. In animals, a reflex 
rule is prewired into the agent, presumably genetically. This means that we can imagine 
the rule as being learned over evolutionary timescales. Animals that had appropriate 
responses (pull your hand away from a burning painful stimulus) were more likely to 
survive and to have children that survived. But fundamentally, the key to a reflex is the 
simplicity of the rule and its prewired nature. 1 

The advantages of a reflex are (l) that it can respond very quickly and (2) because it 
has been learned over evolution, it is present from the beginning in an animal and the 
animal can react correctly the first time it encounters the stimulus. You don’t have to 
learn to pull your hand away from a hot stove. (You really should learn not to touch the 
hot stove in the first place. But that’s a different system.) 

Reflexes are simple things—sensory neurons recognize the stimulus, and these 
neurons connect either directly or through one or more intermediate neurons to 
action-taking neurons that drive the muscles. 2 With vertebrates, this generally takes 
place in the spinal cord because the faster the system can act, the better the reflex evo- 
lutionarily. Remember, reflexes are responding to stimuli already present. That is, your 
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hand is being burned. The longer you wait, the more damage is being done to your tis¬ 
sues. Having the responding neurons be as close as possible to the stimulus allows the 
system to react quickly. Taking the message up to the brain takes time. 

Myelinated nerve fibers carry action potentials at a speed of 150 meters per second. A 
Given that the distance from your hand to your head is about a meter, the journey from 
your hand to your head and back could take 14 milliseconds. Given that your fingers are 
being burned, that’s 14 milliseconds you don’t have, (imagine how long it would take 
a signal to go from the foot to the brain of a giraffe or from the tail to the head of a blue 
whale.) Reflex responses therefore tend to be located in the spinal cord, which is closer 
to the action. 

Reflexes could be simple connections between a sensory neuron and a motor neu¬ 
ron, or even a sensory neuron and the muscle itself. But, in practice, most reflexes have 
intermediate neurons between the sensor and the motor. 4 An interesting question is 
why there is often an intermediate stage within the reflex, given that the time between 
detection and response is so precious. The presence of intermediate neurons provides 
additional stages at which higher, more complex decision-making systems can step in 
to change the responses. 5 This is how we can prevent reflexes from acting if we want. 
Think, for example, of that scene in Lawrence of Arabia we talked about in Chapter 6. 
Normally, Lawrence would respond by reflexively dropping the match or shaking it out 
when the flame reaches his fingers, but with his indomitable will, Lawrence refuses to 
let his reflexes react and holds the match steady. The presence of intermediate neurons 
allows more controls for the other systems to manipulate, but it also increases the time it 
takes the decision to go from sensor to action. Some reflexes do not include intermedi¬ 
ate neurons, while other reflexes have several stages of intermediate neurons. 6 

Classic examples of reflexes are the fleeing reflexes like the wind-sensors in 
insects and the crossover oculomotor turning neurons (visual [oculo-] to motor) 
in the goldfish. 7 Insects such as crickets and cockroaches have a set of very small 
hairs on antennae on their tails. Each hair can be blown by the wind but can only 
move in a single direction, back and forth. This means that the amount that the hair 
moves depends on the direction of the wind—wind that is aligned with the hair will 
move it the most; wind that is off by 90 degrees will move it the least. These hairs 
(wind-sensors) project to four interneurons that represent the four cardinal direc¬ 
tions (ahead, back, left, right). These interneurons project to the leg-muscle control 
structures and make the animal turn away from that wind. Why does the insect have 
this highly evolved wind-response reflex? Because it can flee from a stomping shoe 
in less than 20 milliseconds. 

A similar system exists in goldfish connecting the recognition of visual signals on 
one side with a sudden turn to the other. A goldfish can identify that there is a looming 
signal (such as a goldfish-eating predator, like a diving bird) and start to turn away in 
less than 20 milliseconds. Goldfish can hit speeds of 22 body lengths per second within 


A Neurons communicate information by sending spikes or action potentials along their output 
axons. A nerve fiber (such as the ones that connect your spinal cord to the brain) is just a bunch of 
axons all lining up in the same direction. The speed of travel of these action potentials along a nerve 
fiber depends on several properties, including whether or not the axons are surrounded by myelin, a 
form of glial “helper” cells . 3 
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about 10 milliseconds of movement initiation. That’s incredibly fast! Presumably this 
remarkable reflex evolved because the goldfish and the diving bird are in an arms race to 
see who is the fastest, the hungry bird or the fleeing goldfish. 

Reflexes can, and, of course, do change within a single lifetime. (Animals have 
life-cycles—tadpoles and frogs presumably need very different reflexes to survive in the 
very different environments they inhabit.) But these changes are themselves prewired 
into development. Human babies, for example, have a foot reflex called the Babinski 
reflex, where the toes curl and splay in a specific pattern in response to stroking of the 
bottom of the foot. The reflex is entirely normal (and is used as a check for intactness 
of the spinal cord) in infants. With development of the corticospinal tract (connec¬ 
tions from the cortex in the brain to the spinal cord), the reflex disappears. In adults, 
a Babinski reflex indicates spinal cord damage, decoupling the cortical input from the 
base of the spinal cord. 8 

The problem with reflexes is their inflexibility. Reflexes are learned on an evolution¬ 
ary timescale; they are prewired, genetically, into the creature. Although reflexes do 
change in response to repeated stimuli, the mechanisms of these changes are not par¬ 
ticularly complex, and the potential changes are very limited. These effects have been 
most famously studied in detail in the aplysia gill-withdrawal reflex. 9 An aplysia is a kind 
of sea slug, with a broad, thin gill floating out of its back, (imagine a snail with no shell 
and a pair of silk fans.) It uses this gill to breathe oxygen out of the water. But, of course, 
this gill is fragile and easily damaged. Imagine the aplysia in the water: the waves pick up 
and the aplysia pulls its gill in. But if the waves are going to be large for a long time and 
not get worse, then it needs to open out again. The aplysia can learn to ignore a repeated 
stimulus (that is, it can habituate). However, even these potential changes are prewired. 
For example, the gill-withdrawal reflex in the aplysia can increase its response to certain 
stimuli (that is, it can sensitize), but these changes are due to prewired connections 
between specific neurons. 

Even simple prewired reactions and potential changes can produce extremely com¬ 
plex behavior if the changes interact with the environment appropriately. Examples of 
the complexity that can be built out of simple circuits without memory or explicit repre¬ 
sentation can be found in Valentino Braitenberg’s delightful book Vehicles. Nevertheless, 
this inflexibility means that in the competitive game of evolution, reflexes can get an 
animal only so far. 

While evolution can move populations pretty quickly (on the order of a couple of 
generations),® reflexes are not being learned within a single lifetime. 11 To learn to respond 
differently to different situations, one needs to turn to one of the other decision-making 
systems. 


B A wonderful example of the speed of evolutionary change moving phenotypes within popula¬ 
tions is the black and white moths that changed their distributions in response to smog in industrial 
England, where a population of peppered moths changed from being mostly white to mostly dark as 
coal pollution darkened the skies and trees of England in the late 1800s, and then changed from being 
mostly dark back to mostly white as the pollution was controlled. 10 More detailed examples of how 
quickly evolution can change populations can be seen in the studies by Rosemary and Peter Grant 
measuring the shape and length of beaks of finches in the Galapagos Islands. The Grants’ finch studies 
are described in detail in Jonathan Wiener s The Beak of the Finch. 
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Books and papers for further reading 

Because the reflex is so well understood, it is well described in many medical and neu¬ 
roscience textbooks, such as 

• Dale Purves et al. (2008) Neuroscience. Sunderland, MA: Sinauer Associates. 

A thorough review of the remarkable escape reflexes can be found in 

• Robert C. Eaton (1984) Neural Mechanisms of Startle Behavior New York: Springer- 
Verlag. 

A delightful book showing how complex prewired reactions can be is 

• Valentino Braitenberg (1986) Vehicles: Experiments in Synthetic Psychology 
Cambridge, MA: MIT Press. 


8 

Emotion and the Pavlovian 
Action-Selection System 


see with your eyes 
until they are hands 
hands that feel 
hands that hold other hands 
hands that touch the heart 


The Pavlovian action-selection system consists of response sets 
evolved to prepare for critical species-specific behaviors such as 
feeding, fighting, or mating. Learning in the Pavlovian action- 
selection system entails recognition that one thing likely causes 
another, leading to taking the response set appropriate for that 
expected outcome. Linguistically, we categorize these response sets 
and identify them as emotions. In humans, many of these response 
sets are important to social interactions. 


Although not usually thought of in terms of action-selection, emotions are an 
action-selection mechanism. 1 Each emotion is a response set that has been learned over 
genetic timescales to be useful. For example, in dangerous situations, depending on 
emotional characterizations of the situation, one becomes afraid and one’s body pre¬ 
pares to flee, to stand and fight, or to submit, sometimes referred to as the fight-or-flight 
response (or more accurately, the fight-flight-freeze response). 

There is a debate going on in the scientific community as to whether the categoriza¬ 
tion process of what we consciously label as emotions is consistent between individuals. 2 
These questions turn more on the issue of our conscious labels of emotions rather than 
the physical effect of those emotions—no one denies that anger prepares you for fight¬ 
ing, but other emotions (such as fear) also prepare you for fighting. Even though a given 
situation might make one person angry and another afraid, animals (including humans) 
have self-consistent response sets to consistent situations. We will return to the relation¬ 
ship between these response sets and emotion later in this chapter. 
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The advantage of having a system with these response sets is that there is a complex 
of body-related internal actions that need to work correctly the first time. You don’t get a 
chance to learn to run from the lion. You can learn that a rustle in the grass likely means 
that a lion is stalking you. And so you can learn to run from the rustle in the grass. In fact, 
you can learn that a rustle in the grass means a lion is stalking you even without being 
stalked. You could learn it from observing a lion stalking another animal, or you could 
learn it from hearing a vivid story by the fire, or some other way. In any case, what you’ve 
actually learned is that lions stalking prey produce a rustle in the grass. So when you 
hear the rustling, you realize there’s a lion, and you have the same emotional response as 
if a lion suddenly appeared in front of you. 

What is needed is a system that can recognize the danger and implement all of the 
body changes quickly, all at once. That system is the Pavlovian action-selection system, 
in which a genetically learned action occurs in response to a given situation based on 
an emotional characterization of it ( I like it, I will approach it. I am afraid of it, I will run 
away from it. I am angry at it, I will attack it. I have nowhere to run, I will freeze in fear and 
hope that it won't kill me). 3 

In classical psychology experiments, Pavlovian learning is described as what is 
learned when the animal does not have to take an action in order to get the outcome— 
famously, a bell is rung and the food is delivered, or a light is turned on and then the 
subj ect is given a small shock. 4 There is nothing the animal has to do or can do to change 
these events. In practice, the animal does things to prepare for the expected outcome 5 — 
the dog salivates, the rat freezes. 

At one time it was assumed that the animal was learning that cues co-occur; how¬ 
ever, we now know that Pavlovian conditioning entails recognition of informative 
contingencies—animals learn that when a cue is present, certain things are more likely 
to happen. An ever-present cue is not informative. 6 Animals can even learn that certain 
cues predict that an event will not occur. As the lion example at the start of this chapter 
shows, we are learning that a cue predicts an outcome. (Pavlov’s bell worked this way 
as well—the bell was rung and then the dog was fed.) Stated in this way, the learning 
part of Pavlovian action-selection is that a stimulus (rustle in the grass, ringing a bell) 
predicts an outcome (a lion is stalking you, food is coming). From that prediction, you 
can take the appropriate action (run!, salivate). 

When studied in classical psychology experiments, the action-selection part of the 
Pavlovian action-selection system is often seen as species-specific behavioral responses 
and approach or avoidance responses, 7 but the largest changes are in fact somatic— 
changes in the body itself. 8 In response to fear, for example, the heart rate increases 
(as does the force of each heartbeat) to move more blood to the muscles, the airways in 
the lungs dilate to allow more oxygen, and a host of other responses occur that prepare an 
animal to either fight or run. Attention increases, increasing the ability to sense stimuli. 

One of the best ways to measure an animal’s fear is to measure the startle responses in 
a given situation. A surprising sound literally makes the animal jump; under conditions 
of increased anticipation of danger (fear), the animal will jump higher. By measuring 
the force of the muscles when the animal jumps, one can quantitatively measure its pre¬ 
paredness for action, and its fear. 9 

One does, of course, see the same effects in humans. One time, when I was first set¬ 
ting up my lab, I was worried about whether I could safely plug in a very expensive com¬ 
puter part without turning the computer off first. (The computer was in the process of 
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running a very long computation and I didn’t want to disturb it, but I needed to access 
that part.) The evening before, a new Zip drive (a kind of specialized portable drive 
before USB flash drives were available) had caught fire and burned out at my house 
(with beautiful blue sparks), so I was reasonably nervous about plugging in this very 
expensive part. I expressed my concern to the lab, and decided that I should be the one 
to plug it in so that if something went wrong, it would be my fault. While I was nervously 
waiting to plug the part in, unbeknownst to me, one of my grad students had snuck up 
behind me. The instant I plugged the part in, my student whispered “bzzzzt!” in my ear. 
I must have jumped three feet in the air. 

Another related effect in humans is that we tend to sweat when we are emotionally 
aroused. 10 Because sweat is basically salt water, it conducts electricity, and this is mea¬ 
surable by an increase in skin conductance. Even very subtle changes are detectable as 
a change in the conductivity of the skin. These changes are known as the galvanic skin 
response (GSR) or the skin-conductance response (SCR) and are a surprisingly reliable 
measure of emotional arousal. 

Neurophysiologically, these responses are controlled by small bundles of neurons A 
in the base of the brain, called the hypothalamus,® and by structures at the top of the 
spinal cord, particularly the periaqueductal gray c and the amygdala. 11 

These brain structures interact with the rest of the body through a feedback loop 
between the hypothalamus in the brain and organs in the body via hormones circulat¬ 
ing in the blood. The hypothalamus sends information to (and receives signals from) 
a structure called the pituitary gland, which is a collection of hormone-releasing and 
hormone-sensing cells that translate neural signals to and from hormonal signals. 12 
This pathway from brain to body is actually a complete loop that includes signals from 
the hormones to the rest of the brain as well. Neurons in the brain are cells and thus 
require nutrients, which they get from blood.® These hormones (such as adrenaline, 


A These small bundles are called “nuclei,” not to be confused with the nucleus of the cell. The word 
nucleus (from the Latin word for “kernel” or “nut”) just means a central concentration. In an atom, the 
nucleus (from which we get “nuclear bombs” and “nuclear physics”) is the central concentration of 
particles. In a cell, a nucleus is the central repository for DNA. And in the brain, a nucleus is a concen¬ 
tration of cells that forms an anatomically identifiably separable object. 

B Hypo (below) + thalamus (another part of the brain). 

c A small structure of gray matter (neuronal cell bodies) surrounding a pocket of cerebral spinal 
fluid at the base of the brain ( periaqueductal —near the aqueduct). The periaqueductal gray sits on the 
path of the spinothalamic tract, where cells in the spinal cord send information up to the thalamus and 
thalamic cells send information back down to the spinal cord. 

D A small nucleus (kernel) of cells located between the orbitofrontal cortex and the ventral stria¬ 
tum, shaped like an almond. Amygdala is the Latin word for almond. 

E Actually, the mammalian brain has an intermediate stage between the blood and the brain made 
of a kind of glial cells that filter chemicals in the blood so that only certain chemicals can transfer from 
the blood to the brain. The capillaries are surrounded by extra-tight walls made of these glial cells to 
prevent large things (like bacteria!) from getting through. This is referred to in medical textbooks as the 
blood-brain barrier and is one of the reasons that certain drugs primarily affect the body, while other 
drugs primarily affect the brain. Many of the hormones that we’re talking about do translate across the 
blood-brain barrier, but others don’t. The blood-brain barrier is weaker in parts of the brain where 
these hormone-releasing and hormone-sensing structures five (like the pituitary) and in areas where 
external chemical signals need to reach the brain (such as the olfactory sensors in the nose). 
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testosterone, estrogen, and glucocorticoids) can dramatically affect neural function, and 
even neural structures. 13 The brain is tightly coupled to the rest of the body through 
these neural-hormonal interactions. 

Interestingly, one of these hormone-releasing glands that connect the brain to the 
body is the pineal gland, which was the specific brain structure that the philosopher 
Rene Descartes (1596-1650) identified as the connection between the body and the 
soul. 14 Descartes argued that animals were machines, but humans are not because 
humans have a soul. Descartes argued that the connection between the body and the 
soul lay in the pineal gland. This hypothesis (that the soul and body are different things) 
is called dualism, and is generally phrased in the modern era in terms of a hypothesis that 
the mind and body are different things. Of course, as everyone who has studied anatomy 
knows (including Descartes, who was an exceptional anatomist, well familiar with both 
human and nonhuman anatomy 15 ), all mammals have pineal glands. 16 We will return to 
this discussion at the end of the book, when we directly address dualist and materialist 
philosophies. 

An attentive reader may be wondering at this point in our discussion of “emotion” 
where exactly is the emotion ? Anger, jealousy, lust, fear, happiness, sadness—these are 
feelings-, they are more than a body’s response to stimuli. Emotions are more than the 
fight-or-flight response. The answer lies in the fact that we have also developed a mecha¬ 
nism for monitoring these bodily responses, and we have a mechanism for categorizing 
them. 17 

William James F (1842-1910) and Carl Lange (1834-1900) first proposed that emo¬ 
tions were a connection between the mind and the body. 18 Everyone knows that when 
we get angry, our heart rate increases and our capillaries dilate to allow more blood to 
our muscles, which makes us flush and actually heats our skin. As pointed out by George 
Lakoff and Mark Johnson in their discussion of languages, the metaphors that we use for 
describing emotions reflect the actual bodily changes occurring during those emotions. 19 

As was mentioned when I introduced the situation-recognition system (Chapter 6), 
I suspect that the cortex is one big categorization machine. (A computational discussion of 
the mechanisms of categorization can be found in Appendix C. As discussed therein, the 
interaction between these categorization processes and reality produces categories that 
are often similar but can diverge between individuals. The fact that not everyone agrees on 
the specific boundaries of these categories does not imply that there is not a species-based 
set of responses that are categorized similarly between individuals.) If we applied this cat¬ 
egorization machinery to our internal, visceral sensory system (Howfast is your heart beat¬ 
ing?), we would categorize these Pavlovian responses. Different parts of the cortex receive 
different inputs and categorize different aspects—visual cortex is processing visual input 
because it receives input from the retina (via the thalamus); auditory cortex is process¬ 
ing auditory input because it receives input from the cochlea (again via the thalamus and 
other structures). Is there a cortical area that receives input from the hypothalamus, that 
can recognize these internal states, something that receives visceral inputs? 


p William James is the older brother of the novelist Henry James, who was, one has to say, well 
acquainted with emotions. 
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Anatomically, that part of the brain is the ventromedial prefrontal cortex, G sometimes 
known as the orbitofrontal H cortex, but also including the insula. * 1 These neurons receive 
input from the body to both represent and control the emotional bodily states, a pro¬ 
cess referred to as interoception. 11 This localization of emotional processing was initially 
discovered by studying patients who had damaged prefrontal cortices. 22 These ven¬ 
tral prefrontal cortices have the necessary anatomical connectivity, are known to be 
active during decision-making (as evidenced by fMRl), and play an important role in 
decision-making (as evidenced by lesion studies). 23 

The most famous of these patients was Phineas Gage, a railroad worker who in 1848 
had a tamping iron (a 13-pound, 3-foot-long, 1.25-inch-diameter cast-iron rod) blasted 
through his skull. His job was to tamp (pack) down the blasting powder before it was 
set off. Unfortunately, the process of tamping it down set it off and drove the iron stick 
through his skull. Remarkably, Gage survived; even more remarkably, he recovered 
from his injuries and lived for an additional dozen years. 24 Gage had surprisingly specific 
if complex deficits from his injuries—his physical and sensory abilities, his language 
and intellectual abilities, his ability to perceive the world, his memory, and his ability 
to take action were all intact. What changed, however, was his personality and his abil¬ 
ity to make decisions. Reconstructing the path of the iron bar through a brain sized 
to match Phineas Gage’s suggests a complete destruction of the prefrontal part of the 
brain, including both ventral and dorsal aspects. The dorsal prefrontal cortex seems to 
be involved in the ability to plan, while the ventral prefrontal cortex seems to be involved 
in the ability to recognize emotions. 25 If both of these were damaged, Gage’s ability to 
make decisions, particularly intelligent ones, would have been severely compromised. 

Literature has often described the difference between humans and robots as 
emotional—humans have emotion, robots don’t. From Star Trek’s Data, who spends his 
life wishing he had the emotional interactions that his colleagues do, to the replicants in 
Ridley Scott’s Blade Runner (and in Phillip K. Dick’s original source book Do Androids 
Dream of Electric Sheep?, which is a much better title), who are identifiable through their 
lack of an emotional response, literature has identified the unemotional as “robotic.” In 
fact, in Blade Runner/Do Androids Dream of Electric Sheep?, the replicants are only iden¬ 
tifiable by the fact that when they look at emotional pictures, their eyes do not dilate. 
In Descartes' Error, Antonio Damasio follows the description of Phineas Gage with a 
description of a modern patient (referred to as “Elliot”) with known damage to the 
ventral prefrontal cortex (confirmed by structural M Rl)/ who has a specific deficit in 


G Ventro (toward the base) + medial (toward the middle) + prefrontal (in front of the front). These 
structures are referred to as “pre”-frontal because other anatomists had already named parts of nonhu¬ 
man animal brains as “frontal,” and the anatomists studying human brains wanted to identify an area 
that was even more frontal than the other primate brains. The extent to which nonhuman mammals, 
such as monkeys and rats, have “pre”-frontal cortex is a subject of much debate. 20 

H Orbito (near the eyes—from the word orbit, meaning the circular bone around the eyes) + frontal. 

1 Insula, from the Latin word for “island,” because it sits deep in the folds of the human cortex. 

1 Structural MRI uses a similar technology to the more well-known functional MRI (fMRI) to 
measure the size of gray matter (neuronal bodies) and white matter (neuronal wiring—primarily axons 

and glial support structures [myelin]) in the brain. Because these areas are not changing, it is possible 
to get very accurate pictures of the structure of a brain with structural MRI. 26 
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emotion. Like the replicants in Blade Runner, this patient was blind to emotion: unable 
to process feeling, to recognize it, or to show it. In situations where emotions are all but 
inevitable, Elliot showed no reaction. Damasio quantified this by measuring skin con¬ 
ductance, and found that Elliot’s skin conductance was completely flat. 


How does emotion relate to decision-making? 

Historically, philosophers have suggested that there are two decision-making systems, 
an emotional (hot) decision-making system and a cognitive (cold) system. This hypoth¬ 
esis can be traced back to Plato, who described the human as a charioteer driving two 
horses, one well behaved and always doing the right thing, and the other wild and hard 
to control. 27 * St. Augustine describes two souls, one living in the City of Man, con¬ 
nected to the animals, filled with emotion, lust, and anger, and the other in the City 
of God, separated from the animals, filled with reason and deliberation. 29 (The inher¬ 
ent dichotomy of this is embodied in the wonderful line from Augustine’s Confessions: 
“Make me chaste, O Lord, but not today.” 30 ) 

Freud’s theory of the unconscious follows this same dichotomy. Freud proposed that 
there were three components, the id, the ego, and the super-ego. 31 This tracks Plato’s three 
beings in the chariot—the wild horse that needs to be held in check (the id), the angelic 
horse trying to lead one to reason (the super-ego), and the charioteer trapped in the 
middle (the ego). Much of modern psychiatry still remains within this dual-system L 
hypothesis, based in large part on Freudian concepts of the unconscious. It is, of course, 
as we have seen, fundamentally wrong. You are not separate from your emotional self, 
nor are you a charioteer driving two other decision-makers; you are the sum of all of 
your decision-making systems. 

Modern dual-process theories argue for a separation between an unconscious, 
impulsive (Dionysian) system and a conscious, executive-function (Apollonian) sys¬ 
tem. 32 The typical analogy presented is that of a horse and rider, although some authors 
refer to this as an elephant and its rider. In this analogy, you are the rider trying to steer 
the horse (or the elephant), but the horse (elephant) has a mind of its own and some¬ 
times goes its own way. 

Translating these dual-process theories into our four action-selection systems sug¬ 
gests that the conscious being you are is the Deliberative system, while the Reflexive, 
Pavlovian, and Procedural systems are nonconscious. Even if we assume that there are 
multiple nonconscious systems, this multiple-process theory suggests that the conscious 
self struggles to control the other systems. This differentiation between the noncognitive 


K This is a great example of the homunculus fallacy, in which the decision-maker is seen as a little 
person (a “homunculus”) controlling a separate body or decision-making system. 28 The problem, as 
pointed out by Daniel Dennett in the 1980s, is that someone will then need to explain what controls 
that driver, leading to an infinite regress. One of the points of this book is that you are your brain— 
there is no separate homunculus driving; you are all of these decision-making systems, interacting with 
each other. 

L The fact that this is always referred to as a “dual-system” hypothesis emphasizes the danger of the 
homunculus fallacy—dual-system theories forget that Plato’s two horses require a driver. 
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(unconscious) other and the cognitive (conscious) self requires that we remove emo¬ 
tion from ourselves and place it in the “other” box. I don’t know about you, but I have 
always felt (an emotional word!) that my emotions are a critical part of who I am. 

Separating the cognitive “self” and the noncognitive “other” also requires that we 
separate out the Procedural (habit, cached-action) system as part of the “other.” The 
Procedural system is not emotional in any sense of the word—it is a simple storage of 
stimulus-response pairs and response-response action chains. Instead of the two pro¬ 
cesses inherent in dual-process theory (emotional, unconscious, the horse or elephant 
vs. cognitive self, the rider), we would have to separate three systems (emotional, 
procedural, self). But, again, the procedural responses are part of us. We will see that 
these stimulus-response pairs and these action chains are the key to athletes hitting 
baseballs, making plays, throwing and catching footballs, and making other trained 
responses. When asked, most athletes talk about coming into themselves, not going out 
of themselves, when they are “in the zone.” 

As pointed out by Antonio Damasio in several books, people like “Elliot” with 
impaired emotional systems do not make normal decisions. 33 In fact, there is now 
some evidence that people who are psychopathic or sociopathic are calculating and 
unemotional, making decisions independent of any emotional sensitivity. 34 

When talking about emotions and the role of emotion in decision-making, it is 
important to note that our memories of the past are reconstructed, rebuilt each time 
anew and are not necessarily correct. (We will address this issue in depth in our dis¬ 
cussions of episodic future thinking, episodic memory, and deliberation in Chapters 9 
and 16.) This means that if we took an action based on one of our other systems—for 
example, because of our Procedural learning system—we may imagine that we took that 
action because our Deliberative system wanted to, or we may imagine that we took that 
action because of an emotional response. Although one may remember taking an action 
“because it felt right,” 35 in fact, the emotional parts of our brain that actually encode 
“feeling right” might not have been active at all at the time; instead, you may have 
back-constructed the emotion. 36 This has been explicitly seen in studies of craving and 
relapse to drug addiction: most addicts will report that they remember an unstoppable 
craving in the days before relapse, but prospective studies that look at addicts during 
relapse find that only a small proportion actually show craving. 37;M 

That being said, emotions do play a strong role in our decision-making. Often, sit¬ 
uations in the world or situations with our fellow humans produce emotions, which 
guide our decisions. Emotions can be described as our interpretation of reactive states 
that have been learned over evolutionary timescales, producing changes in our bodily 
responses and biasing our actions. 

When walking along a dark road at night, sounds begin to make us nervous, and we 
begin to think we may be seeing things. The developing anxiety enhances our attention 
to the point where we may even see things that aren’t there. I have not gone into depth 
about perception in this book, in large part because so much is known about it that 


M Prospective studies are, of course, a lot harder to do because they have to look at many more sub¬ 
jects to find the ones who will relapse naturally during the course of study, and because they have to he 
constructed very carefully so as not to prime those subjects to start craving or to relapse (which would 
be unethical and would be looking at cued relapse rather than natural relapse). 
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I would need an additional book to explain it all. 38 Two important concepts, however, 
are useful here. First, perception is fundamentally an interpretive action. 39 Although 
our retina responds to light, we do not see “light,” we see objects—trees, houses, faces. 
Although the neurons in our cochlea in our ears respond to sound, we do not hear 
“sound,” we hear voices and music. Illusions occur because the brains interpretation 
mechanisms are being tricked and process information incorrectly. Second, perception 
is fundamentally about detecting signals. 40 This means that one has to make an estimate 
of whether the perceived set of light signals is a face or something else. When I look at a 
tree in broad daylight, it’s pretty easy to realize that’s a tree. But late at night, in the dark, 
is that a tree waving in the wind or something more sinister? 

One time, shortly after we graduated from college, my wife and I were visiting Ireland. 
We were on the Aran Islands, a very rural part of the westernmost part of Ireland. We 
were walking back from the pub to the bed-and-breakfast where we were staying. It was 
pitch black except for the barest of starlight peeking through the quickly moving clouds. 
Well versed in Irish poetry and Irish myth, it was pretty easy to get spooked by the dark¬ 
ness. By the edge of the road ahead of us, we saw this orange light floating in the air, hke 
a will-o’-the-wisp. There was this smell in the air, hke a dog or a wolf. By the time we got 
to the light, our adrenaline was at full strength and I was ready to jump. As we passed, a 
young man, sitting on his front stoop, smoking a cigarette, his hand on the head of this 
big, furry dog, nodded to us, said “Evenin’,” and asked us where we were heading. We 
told him and he said it was about a mile down the road. And we went on, laughing at our 
emotional interpretations of the floating orange light. 

In signal detection, there are two types of errors one can make, false negatives and 
false positives. A false negative entails not recognizing something that is there, while 
a false positive entails recognizing something as being there when it is not. Most 
signal-detecting mechanisms have parameters (gains and thresholds) that can be 
adjusted to trade off the likelihood for making false-positive errors from false-negative 
errors. 41 The brain is no exception. 

This means that perception, hke any signal-detection machinery, has to trade off false 
negatives (missing something that is there) for false positives (identifying something 
as being there when it is not). As we become anxious, even frightened, by the dark, our 
hearts start to pound, our muscles become tense, ready to spring into action, and our 
attention looks for anything that might demand a response. At the sound of a branch 
cracking beneath our feet, we might literally jump in the air. (This is fear-potentiated 
startle, which we have seen before.) 

Emotions are particularly important for social interactions. Love at first sight may or 
may not exist, but lust at first sight certainly does. A person who responds to a situation 
(say catching a cheating spouse) with anger and rage is going to select fundamentally 
different actions to take than someone who responds to the same situation with sadness 
and dejection. In a now-classic example, people offered unfair rewards in the ultimatum 
game N show increased activity in the anterior insula (involved in negative emotions) 
and reject the unfair offer with anger and disgust. 43 


N We will discuss the ultimatum game in detail in our discussion of morality. Briefly, the ultimatum 
game entails two players dividing up a sum of money, say $20. One player gets to make an offer of 

( continued ) 
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Studies of emotion have found that not only are many decisions correlated with emo¬ 
tional reactions to the decision itself, but also that unrelated emotional states can affect 
decisions. 44 Inducing a feeling of anxiety before being faced with a decision makes peo¬ 
ple less willing to take risks; inducing a feeling of anger makes people less likely to delib¬ 
erate over decisions and more likely to respond quickly. 45 This observation (that being 
emotionally frightened, angry, or sexually aroused makes one less likely to deliberate 
cognitively) is almost certainly where Plato, St. Augustine, and Freud got their dichot¬ 
omy between emotion and cognition pulling the person in two directions. However, as 
I hope I’ve made clear, the interaction is much more complicated than that. 


Do animals show emotion? 

Emotion is often seen as part of our “animal” heritage, while cognition is seen as a 
“higher,” “more human” decision-making system. 46 As I hope I’ve shown throughout 
this book, that’s a mistaken dichotomy. Rats deliberate over decisions, and emotions are 
fundamental to the interactions that make us human. A question that arises, however, 
is whether animals have emotions and feelings. All pet lovers, particularly those with a 
mammalian pet (a cat, a dog), will tell you that their pets have feelings, but of course, as 
we saw in the very beginning of the book, we are notoriously bad at judging the agency 
of nonhuman creatures. My father is convinced that his GPS gets annoyed when it has 
to recalculate a route because he missed a turn. 0 On the other hand, apes who have been 
taught limited language (such as Kanzi the chimpanzee or Koko the gorilla) are able to 
express emotions in words. 47 These apes use emotional words in very parallel ways to 
humans. In his book The Expression of the Emotions in Man and Animals, Charles Darwin 
noted that certain emotional expressions cross cultures and species. Anger, disgust, 
laughter, joy, sadness: these are all consistent signals across all humans and all cultures. 

In fact, as we saw in Chapter 4, these facial expressions are similar across mammals. 
Kent Berridge and his colleagues studied facial expressions of distaste (in response to 
bitter liquids) and appeal (in response to sweet liquids) and found similar facial expres¬ 
sions across rats, monkeys, and humans (sticking one’s tongue out, making a “yuck” face 
for bitter and licking one’s lips for sweet). Berridge and colleagues were able to show that 
these expressions are not about taste on the tongue because they could change distaste 
to appeal and vice versa by manipulating the opioid system in rats. 48 


how to split the money, and the other gets to decide whether to take the offer (and both players get 
the offered split) or to reject the offer (and no one gets anything). Notice that a recipient rejecting an 
offer is losing money to punish the person splitting the money. This is a phenomenon called “altruistic 
punishment,” because punishing unfair offers leads to more fair societies, but the person punishing the 
unfair offer has also lost money. 42 We will explore altruistic punishment and its effects on societies in 
our chapter on morality (Chapter 23). 

° Of course, he knows that the GPS is simply using an algorithm for identifying when to recalculate 
and playing a voice-generation file to report it. (In fact, he could probably program the algorithm.) 
Nevertheless, our emotional responses to social situations (such as hearing a recorded voice of an exas¬ 
perated person saying “recalculating”) is evolutionarily prewired into our brains. 
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An interesting question, of course, is where emotion arises in the evolutionary time¬ 
line. Do bacteria have emotions? Probably not as we understand them. Do reptiles? 
Unlikely. Again, scientists have not been able to reliably identify them. Do mammals? 
Almost certainly. One intriguing proposal put forward by Michel Cabanac, Arnaud 
Cabanac, and Andre Parent is that amniotes (mammals, birds, and some reptiles) devel¬ 
oped a categorization system that can observe the body’s Pavlovian action-selection 
system and that this categorization system is what we now recognize as emotion. 49 This 
is consistent with William James’ concepts of emotion as actually residing in visceral 
changes in the body and Antonio Damasio’s concepts of emotion as the recognition and 
memory of bodily responses, which he refers to as the somatic marker hypothesis . 50 

Whenever emotions appeared evolutionarily, they are clearly an important part of 
human decision-making systems. Emotions need to be seen not only as feelings that 
drive other decision-making components to act, but as action-selection systems in 
their own right. Emotions interact with our categorization of feelings through visceral 
changes in the body, which we identify as those same emotions through memory and 
metaphor: the vigilance of fear, the tension of anger, the heat of lust, the warmth of 
comfort, the breaking-heart pain of sadness. 

Books and papers for further reading 

• Antonio Damasio (1994). Descartes' Error: Emotion, Reason, and the Human Brain. 
New York: Quill Press. 

• Antonio Damasio (2003). Looking for Spinoza: Joy, Sorrow, and the Feeling Brain. 
New York: Harcourt. 

• Joseph E. LeDoux (1996). The Emotional Brain. New York: Simon and Schuster. 

• George Lakoff and Markjohnson (2003). Metaphors We Live By. Chicago: University 
of Chicago Press. 
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Deliberation 


Swallowing the leaf-green light, 
stretching under gravity, tension across 
the surface enfolds us into a moment, 

until, suddenly, 
disturbed by a breath, 
we fall. 


Some decisions are made by explicitly imagining the potential conse¬ 
quences of that decision. This imagination process is particularly good 
for decisions that cannot be learned slowly over time, or for major 
decisions that require a large upfront commitment cost. Deliberation 
is a computationally expensive process that requires a number of 
neural systems, including attention, imagination, and evaluation pro¬ 
cesses, and as such takes time to calculate which action to take. 


Deliberation entails the sequential, serial search through possibilities. It entails a predic¬ 
tion of “what would happen if,” followed by an evaluation process and an action-selection 
process. This means that deliberation requires a host of complex processing. This com¬ 
plex processing makes deliberation incredibly flexible (knowing that a path leads to an 
outcome does not imply that one must take that path to that outcome) but also makes 
deliberation slow (it takes time to do all that computation). 

Deliberation also requires a knowledge of the structure of the world, particularly 
the cause-effect structure. 1 (if you can’t predict, you can’t search through those pre¬ 
dictions.) Deliberation requires imagination. 2 Deliberation requires one to be able 
to construct possible futures, and a process to know that those potential futures are 
imagined and not real. A Deliberation also requires an active memory process capable of 
remembering what paths you’ve searched through already and what paths are still left to 


A The inability to differentiate imagination from reality is, of course, a key component of several 
psychological disorders (such as schizophrenia). 3 
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explore. 4 When comparing options, you need to be able to keep multiple options, their 
evaluations, and the paths to them in your head. By attending to specific pros and cons, 
one can even manipulate the evaluation. 5 This requires a process called working memory, 
in which concepts are held in an active processing loop. 8 


The cognitive map and declarative memory 

The concept of the cognitive map was introduced by Edward Tolman in the 1930s. 
Tolman suggested that both rats and humans (and presumably other mammals as well) 
maintained a “map of possibilities.” 11 This meant that learning didn’t have to be aimed 
at getting rewards and avoiding punishments (as suggested by his contemporaries, such 
as Clark Hull and B. F. Skinner 12 ). Tolman’s concept implied that it was worth it to an 
animal to learn information about the world even when that information didn’t imme¬ 
diately lead to a reward or punishment. Information about the world could be used in 
future decisions, making basic information a worthwhile pursuit itself. 

In modern neuroscience, the cognitive map is usually described as if it is a spatial 
map, but Tolman’s original concept was more “cognitive” than “map,” and more mod¬ 
ern interpretations of cognitive maps are closer to the concept of a schema about the 
structure of the world. 13 Certainly, one of the most important things to know about 
the structure of the world is where things are and how the spatial world is connected. 
Knowing that there are skyways connecting the building my office is in and the mail- 
room will be very useful today when it is going to be 2° Fahrenheit outside. c But it is also 
part of the cognitive map to know that one can check books out of a library and there¬ 
fore one doesn’t need to buy them. And to remember a friend’s phone number, or that 
one can call 911 in an emergency (in the United States). Or that Paris is in France and 
Tokyo in Japan, that New York is on the east coast of the United States and San Diego 
is on the west coast. All of this knowledge about the structure of the world forms your 
cognitive map. If you’ve already learned these facts, you can use them to search through 
options to find the choice you want to make. 

These facts are sometimes referred to as declarative 14 because they are things that 
can be “declared”—if I tell you that Tokyo is the capital of Japan, or that I grew up near 


B Working memory keeps ideas, concepts, and results easily accessible for immediate processing. 
The complexity and the mechanisms that allow working memory are a very large topic and are the 
subject of multiple books. 6 It is related to the concept of rehearsal, where one repeats the thing-to- 
be-remembered over and over again, 7 and prospective memory, where one has to remember to be ready 
for a future event 8 (“Don’t forget the milk”). The capacity of working memory is strongly tied to IQ, 
mental ability, success in life, and self-control. 9 Recent experiments have suggested that it is possible to 
train working memory, and that this training has effects on decision-making. 10 

c For my metric-using readers, that’s -17° Celsius! The University ofMinnesota has most of its build¬ 
ings connected by skyways, bridges between buildings set above the street, usually on the second or 
higher stories. These skyways are often beautiful, with large glass windows, but they form a complex 
maze. Some interbuilding connections are above the street, while others are tunnels below the street. If 
you know the secret passages, you can often get quite a ways across campus without ever going outside. 
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Washington DC but that I live in Minnesota, you know those facts. You don’t need to 
practice them. In contrast, I can tell you how to throw a baseball or ride a bike or play 
the piano or clarinet, but to become any good at it, you need to practice them. In the 
memory literature, these other skills are called procedural because they require memo¬ 
ries of the process and procedures rather than facts. 

Declarative memory is often divided into two subcategories: semantic memory 
and episodic memory. 15 Semantic memory entails the facts and knowledge about the 
structure of the world. Episodic memories are the memories of your own life. The 
statement that at my sixth birthday party, my parents made me a cake in the shape 
of a rotary-dial telephone is a semantic memory. My memory that my dad said I had 
to wait to get the last piece because my initials were in the center (presumably try¬ 
ing to teach me to be polite and let my guests have cake first), and that I snuck my 
finger in and ate off the initials while my dad was serving a piece to someone, and 
then demanded a piece of cake since I had already got the letters, is an episodic 
memory. 

The key evidence that animals (including humans) use a cognitive map in making 
decisions is a process called latent learning. Learning about the structure of the world 
may not show up in behavior until one needs to call upon that knowledge. Tolman’s 
initial hypothesis came from observing rats running mazes. 16 Rats with experience on a 
specific maze (just exploring, with no rewards on the maze) would subsequently learn 
much faster than rats with no experience on that maze. The experiment that finally con¬ 
vinced Tolman that animals had to remember knowledge about the structure of the 
world was that a rat (who was neither hungry nor thirsty) who was allowed to explore a 
Y-shaped maze with food on the left fork and water on the right would immediately go 
to the left fork when made hungry and to the right fork when made thirsty. For humans, 
at least, this latent learning is about cause and effect. We recognize this knowledge in 
terms of if-then rules. If I drop this glass, it will break. If I drop this plastic bottle, it 
won’t. Therefore, it is safer to give a plastic bottle to my three-year-old daughter since 
she may well drop the glass. 

We can think of deliberation as having four parts: the knowledge of the world, the 
search process using that knowledge of the world to predict what will happen, an eval¬ 
uation process measuring how good that outcome is, and, finally, an action-selection 
process to actually take the first step forward. 


Search 

The first attempts at building artificial decision-making systems were based on search 
processes. 17 Allen Newell andHerb Simon’s General Problem Solver (originally proposed 
in 1957) searched through possibilities to find a path from a starting point to a goal. 
It was able to create subgoals in the plan, and then solve those subgoals. It worked by 
searching through potential subgoals and then solved each subgoal by searching through 
known actions until it was able to find a path from the starting point to the final given 
goal. Over the past 50 years, a tremendous amount of work has been done improving 
search algorithms to the point that highly efficient search algorithms can beat humans 
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at complex games (like the now-famous Deep Blue that was able to beat the grandmaster 
Garry Kasparov at chess in 1997). 18;D 

Over the past few decades, however, it has become clear that these highly efficient, 
optimized search processes are not how human experts solve most of these tasks. 19 
Search is something done early in learning and in response to changing situations. The 
more expertise people gain in a field, the smaller the set of futures they search through. 
Current theories suggest that this is because experts have cached (stored) the better 
choices ahead of them. 20 Search requires holding a lot of temporary results in memory 
and keeping track of a tremendous number of subparts. This means that it is effortful 
and requires extensive computational processing. 21 

As it became clear that human experts do not solve games like chess by deep searches 
through long possible paths, it became popular to say that search was uniquely human. 
These hypotheses suggested that search was difficult to do, that humans could do search 
when they needed to, but that it required complex cognitive and conscious processes. 
In particular, these theories suggested that search required a linguistic description of 
the problem (that people needed to talk to themselves as they searched through those 
possibilities). This implied that it is a uniquely human characteristic, and that animals 
wouldn’t or couldn’t do search. 22 

On the other hand, it had been known for many years that animals could make deci¬ 
sions based on expectations of the outcomes of their actions. 23 Given the simple pro¬ 
cesses that were hypothesized at the time, it was assumed that these expectations formed 
from simple associations between the action and the outcome. However, computational 
studies of decision-making that began to be developed in the past decade led to the real¬ 
ization that those simple associations are insufficient to solve the problem. Although one 
can solve a one-step decision (approach Cue One when hungry, Cue Two when thirsty) 
with a simple association (Cue One has food, Cue Two has water), if the recognition of 
expectations must cross choices (go left, then right, then left again to get to the food), 
then the only way to solve this is with processes that encode the entire cause-and-effect 
structure of the situation. 24 Thus, several authors have now suggested that animals are 
actually doing search-and-evaluate processes under certain conditions. 25 

In 2007, Adam Johnson and I found that neural signals in the rat hippocampus 
played out representations of future paths at the times when the computational theories 
predicted that animals would be searching through the future. 26 Hippocampal cells in 
the rat encode the location of the animal on a maze. 27 This means that if you record 
from enough cells simultaneously, you can decode that representation. 28 Adam devel¬ 
oped a new mathematical technique 29 that allowed us to decode these representations at 
very fast timescales (cognition happens faster than behavior 30 ), and when we looked at 
movies of that decoded representation over time, we found that sometimes the animal 
would pause at a difficult decision point and those representations would sweep down 
the possible paths, first one way and then the other.® 


D Interestingly, although search algorithms seem to be able to reliably beat human players at chess, 
other games, such as the game of Go, have been much more difficult. 

E We weren’t looking for this. We were actually trying to examine hippocampal representations on a 
specific maze to compare them with our dorsal striatal recordings from rats running that same maze. 31 
But one day, Adam came into my office and said, “Dave, my animals are showing mental time travel!” 

( continued ) 
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In humans, deliberation depends on the process of imagination, in which one creates 
a mental image of a potential future. 32 Whether these sequential sweeps of representa¬ 
tion down the potential paths ahead of the rat that we saw are the same phenomenon as 
a human consciously thinking about the future is still an open question, but the neural 
structures involved match remarkably well. We know, for example, that humans with 
hippocampal lesions have difficulty imagining the future, and that episodic future think¬ 
ing in humans produces hippocampal activity as measured by fMRI. In humans, con¬ 
sciously deliberated decisions depend on prefrontal structures coupled to hippocampal 
systems. In rats, similar prefrontal structures become functionally coupled to hippocam¬ 
pal systems during deliberative decision-making. 33 As we will see in the next section, the 
same structures (orbitofrontal cortex and ventral striatum [nucleus accumbens], both 
functionally coupled to the hippocampus) are involved in evaluation processes in both 
humans and animals. 34 


Evaluation 

Current economic and computational neuroscience models of deliberation do not actu¬ 
ally do much “deliberation.’^ That is, even the ones that include search processes simply 
look up the value of that outcome as a single number, expressed in a common cur¬ 
rency. Obviously, if you are going to compare two choices (such as eating an apple or an 
orange), you have to put them onto the same scale so you can compare them. Economic 
models (and the computer models on which current neuroscience models are based) 
call this same scale a common currency. 37 However, this is not our conscious experience 
of deliberation; our conscious experience of deliberation is that we evaluate the out¬ 
comes in relation to each other. G 


To put it politely, I didn’t believe him; I was deeply incredulous. As we checked and rechecked our find¬ 
ings for possible alternative hypotheses, we were eventually left with the conclusion that our rats were 
indeed representing possible paths ahead of themselves. This is the fun of science—discovering some¬ 
thing completely unexpected. This is captured elegantly in the famous quote usually attributed to Isaac 
Asimov, “The most important phrase in science is not actually ‘Eureka,’ but rather ‘Hey, that’s funny.’” 

F One exception is the work on race-to-threshold models, which accumulate evidence for indi¬ 
vidual choices until one of the choices reaches a threshold. 35 Although there is neural evidence for 
these race-to-threshold models, the evidence tends to come from experiments that include perceptual 
ambiguity, but which have not incorporated the multiple decision-making systems known to exist. 
One experiment has looked at the effect of value ambiguity and found evidence for race-to-threshold 
integration in humans making decisions between equally valued but different foods. 36 We will examine 
race-to-threshold models in our discussion of the role of perception in decision-making (Chapter 11). 
An interesting possibility is that evaluation uses the same mechanisms as perception, applied to 
hypotheses about future internal states. 

G It is worth taking a digression to talk about introspection and psychology. Introspection is the 
conscious internal examination of our own mental processes. It has a very long tradition in psychol¬ 
ogy, going back to the earliest philosophy—the famous dictum was “Know thyself.” 38 The problem 
is that, as we all know (and as much of what this book is about), we are often wrong about our own 
decision-making processes. 39 


( continued ) 
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Introspective and casual observations of the situations where we deliberate over 
decisions suggest that they tend to be more about difficulties in the evaluation step 
than the future prediction step. Much of the computational literature has spent its time 
working on the prediction step of decisions, 44 likely because it arose out of the robotics 
literature, where one needs to have the robot take each step. But, as noted in the earliest 
search models (such as Newell and Simon’s General Problem Solver), we don’t spend 
our time working out the path to the outcome; instead, we create subgoals. 45 

For example, if we imagine a person with two job offers, the action needed is simple— 
send an email to one job saying “yes” and another to the other job saying “sorry.” This 
might engender a long series of subsequent actions (signing a contract, finding a house, 
moving), but our hypothetical applicant doesn’t need to consider how to get to the job 
in order to evaluate the two possibilities, only to know that it is possible to move to wher¬ 
ever the job is. The complexity in determining whether subgoals matter is that traversing 
the subgoal can matter in some conditions, but not others. If I’m deciding whether to 
take the train or the airplane from Washington DC to New York, getting to either the 
train station or the airport is easy (they’re both on the Metro). However, if I’m consid¬ 
ering whether to take the train or the airplane from Baltimore, traversing the subgoal 
becomes an issue because the train station is in town, while the airport is a long ways out 
of town. This issue of how to identify when one can skip the subgoal step and when one 
needs to include it is a complex issue that is the subject of much current research. 


Normal psychology (the study of normal mental processing, as compared to “abnormal psychology,” 
the study of mental disorders) arose in the 1800s with the study of perception. 40 These experimental 
psychologists developed a technique in which they trained themselves to respond to signals quickly, and 
quantitatively measured how quickly they recognized something (such as a picture of a face). 

The psychologists using introspection at the end of the 19th century called themselves Gestalt psy¬ 
chologists, from the German word Gestalt, meaning form or shape, because they believed that they 
were learning to recognize the overall form or shape of the world. The problem was that they were mea¬ 
suring introspective responses. (How quickly do I recognize this signal?) With the discovery by Sigmund 
Freud (1856-1939) that some psychology was unconscious, the conscious-based experiments of the 
Gestaltists became problematic. (It is important to recognize that the Gestalt experiments were quan¬ 
titative [measuring how quickly they responded to a signal], even if they were introspective.) In the 
early part of the 20th century, these introspective experiments were abandoned for direct measures 
of behavior, under the guise of behaviorism, which led to the belief that since it was impossible to 
measure internal cognition, introspection was not good scientific practice. 41 The argument was that we 
should treat the brain as a black box, since the only thing we can measure is behavior. Of course, as we 
have seen throughout this book, we can now open the black box and measure things other than behav¬ 
ior (such as neural activity; see Appendix B). We can now measure those internal cognitive states 42 
Nevertheless, the problems with introspection remain. 

The fundamental problem with introspection, of course, is that, as we saw in Chapter 1, we con¬ 
tradict ourselves, we each contain multitudes, and not all of our internal processes are accessible to 
our consciousness. 43 This doesn’t mean that we can’t use introspection to guide our scientific hypoth¬ 
eses, but we are going to need to check those hypotheses with careful experiments based on direct 
observations. In psychology, this is called operationalizing our hypothesis—the idea is that we can’t 
measure introspective concepts directly, but we can measure behavior (or neural signals). If we can find 
something that correlates with or differentiates our introspective hypotheses, then we can scientifically 
study that. 
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In any case, once we have identified the potential outcomes, we need to evaluate 
them. That evaluation step is complex, depends on attention to details, and is subject 
to interesting framing effects. 46 Although the mechanistic process by which we evaluate 
these multiple noncomparable objects is currently unknown, we know that this process 
is affected by the other decision-making systems, particularly the emotional system. 47 
It’s not surprising to anyone who has observed his or her fellow human beings, but tests 
have shown that your emotional state at the start of a decision can influence your ulti¬ 
mate deliberative decision. 48 At the most trivial level, we can see this in Tolman’s original 
rat experiment—hungry rats went to the food, thirsty rats went to the water. 49 But this 
also occurs at more complex levels—for example, people who are in an angry or sad 
mood are more likely to reject offers in trading games. 50 

We also know some of the brain structures involved in these evaluation processes, 
because we know what happens when these structures break down. Two structures 
known to be deeply involved in evaluation in general and in deliberative evaluation in 
particular are the ventral striatum and the orbitofrontal cortex. 51 The ventral striatum, 
sometimes called the nucleus accumbens, sits at the bottom of the striatum, which is a 
large, evolutionarily old, subcortical structure. 61 The orbitofrontal cortex sits at the bot¬ 
tom of the front of the cortex, just behind the eyes. * 1 When these structures are damaged, 
people and animals have difficulty evaluating options. 53 

Neural firing in both of these structures relates to value judgments of outcomes. Both 
structures contain some cells that respond to reward consumption, as well as other cells 
that respond to cues that predict rewards, and other cells that seem to represent the 
expected value of an outcome. 54 This has been most explicitly tested in monkeys and rats 
making economic decisions between objects. For example, Camillo Padoa-Schioppa 
and John Assad gave thirsty monkeys decisions between two flavors of juices. By vary¬ 
ing the amount of juice offered, they could determine how much each juice was worth to 
the monkey, (is one drop of grape juice worth two drops of apple juice?) What they found is 
that some cells in the orbitofrontal cortex represented how valuable the apple juice was, 
other cells represented how valuable the grape juice was, and some cells represented 
how valuable the choice the monkey made was, translated into a common currency, 
independent of whether it chose apple or grape juice. Similar effects have been found in 
humans using fMRI experiments. 55 

Orbitofrontal neurons also reflect framing effects. In tasks where all of the choices 
are interleaved, the neurons correctly reflect the revealed preference value between the 
choices. 56 In tasks where the options are given in groups, the cells reflect the valuation 


H Ventral means “toward the chest” in Latin and can be contrasted to dorsal, which means “toward 
the back” in Latin. In animals that run on all fours (quadrupeds, like dogs and cats), the chin of the ani¬ 
mal is toward the chest, while the top of the head is toward the back. In humans, because we’ve stood 
up (become bipedal), the bottom of the brain is toward the ground and the dorsal part of the brain is 
toward the sky. Scientists stick to these terms so that they can use the same terminology for the same 
parts of the brain, such as the dorsal and ventral striatum, which are well conserved between other ani¬ 
mals and humans. 52 Think of ventral as toward the bottom of the brain and dorsal as toward the top. 

1 Thus orbito, from the “orbit” of the eyes, referring to the circular bone that surrounds the eye, 
and frontal. We encountered the orbitofrontal cortex in our discussion of emotion and Pavlovian 
action-selection systems (Chapter 8). 
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between the available options. Leon Tremblay and Wolfram Schultz gave a monkey 
choices between raisins and apples or apples and cereal. From previous experiments, 
they knew that the monkey they were recording from preferred raisins to apples and 
apples to cereal. They found that value-encoding neurons fired very little to the apples 
in the raisin-apple comparison but a lot to apples in the apple-cereal comparison. In 
contrast, Camillo Padoa-Schioppa and John Assad did not find these changes to their 
three juice options, but the Padoa-Schioppa and Assad experiment interleaved the 
choices, while the Tremblay and Schultz experiment grouped the choices. This grouping 
is called “blocks” in the jargon of behavioral neuroscience. That is, a subject may receive 
a block of choices between apple and grape (say for 50 trials) and then another block of 
choices between orange and banana (say for 50 trials) and then another block of apple 
and orange... etc. In subsequent experiments, Padoa-Schioppa found that the activity of 
orbitofrontal cortex neurons depended on exactly this issue —What is the available menu 
of choices? Just like humans making choices between televisions determine the value of 
the choices relative to the other options^ monkeys making choices between foods or 
juices determine the value of the choices relative to the other options available. 

Of course, these preferences can change. If you get sick after eating something, you 
will learn to dislike that food. 58 (This is why chemotherapy patients are given strange 
flavors of foods, so that the extreme nausea that chemotherapy induces does not teach 
them to dislike their favorite foods. 59 ) Similarly, overexposure to a given food flavor can 
produce satiation, and preference for other foods. 60 As an animal changes its preference 
(say because it gets satiated because it has had enough of the apple juice, or because 
it has a bad experience after drinking that apple juice), the orbitofrontal cortex neu¬ 
rons change to reflect that changed preference. 61 One of the effects of drugs of abuse, 
particularly cocaine, is that they disrupt the ability of orbitofrontal cortex neurons to 
recognize changes in value. 62 Orbitofrontal neurons in animals with extensive cocaine 
experience do not successfully change valuation when the action that leads to a sweet 
flavor is changed to now lead to a bitter flavor. Part of the problem with drugs is that they 
affect the ability of the brain to correctly re-evaluate situations when they change. 

In a very interesting recent experiment, Geoff Schoenbaum and his colleagues tested 
whether rats could recognize a change in the reward that would occur after taking an 
action 63 (either by increasing the number of food pellets provided, say from one food 
pellet to three, or by changing the flavor of the food provided, say from banana-flavored 
to cherry). If the change in value was numeric (that is, changing the size but not the 
flavor of the reward), the animal needed the ventral striatum intact, but not the orbitof¬ 
rontal cortex. However, animals needed both the ventral striatum and the orbitofrontal 
cortex to recognize the change between flavors. This suggests the ventral striatum is key 
to recognizing changes in valuation, and the orbitofrontal cortex to recognizing changes 
in commodity. 

So the data suggest that these structures are involved in evaluation, but are they active 
during the evaluation step in deliberation? fMRI data strongly suggest a role for both 


Remember the television example from Chapter 3. When given a choice between $100, $200, and 
$300 televisions, people are much more likely to pick the $200 television over the $300 one. But when 
given a choice between $200, $300, and $400 televisions, people are much more likely to pick the $300 
television over the $200 one. This is called framing, because the set of choices frames your decisions. 57 
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the orbitofrontal cortex and the ventral striatum during some aspect of deliberation. 64 
Direct evidence for a role for these neural structures in evaluation during deliberation 
comes from neural recording data taken by my former postdoc Matthijs van der Meer 
and by my graduate student Adam Steiner. 65 As we discussed earlier, at certain difficult 
decisions, rats pause and look back and forth (a phenomenon known as “vicarious trial 
and error”). 66 We have already discussed how, during this process, the hippocampus 
of rats represents potential choices ahead of the animal, as if the animal was searching 
down possible paths. 67 When he was in my lab, Matthijs van der Meer recorded from 
ventral striatal neurons from rats on this same task and found that at the same times 
that Adam Johnson had found the hippocampus sweeping through those future paths, 
the ventral striatal reward cells (cells that responded positively during consumption) 
activated again, as if the ventral striatum was covertly representing something about the 
fact that the animal would get reward if it went in that direction. Recording orbitofron¬ 
tal cortex neurons, again, from the same task, Adam Steiner found that orbitofrontal 
reward cells (cells that responded positively during consumption) also activated, again, 
as if the orbitofrontal cortex was covertly representing something about the fact that 
the animal would get reward. One interesting difference we found, however, is that the 
ventral striatal cells reflected the expectation of reward before the animal turned toward 
that reward (suggesting involvement in the action-selection), while orbitofrontal cor¬ 
tex cells were active after the animal had already turned toward the reward (suggesting 
involvement in the prediction of an outcome more than the action-selection). 

The evaluation step in deliberation is still an area of active research both in humans 
and in nonhuman animals. We know some of the structures that are involved, and some¬ 
thing about the timing of the mechanisms, but we still don’t understand how attention 
changes the framing of the evaluation process. For example, when comparing cars, 
imagine trying to decide between a sports car and a minivan or hybrid: one is snazzier, 
one can carry more, one gets better gas mileage. There is no direct way to compare snazz, 
size, and gas mileage on a “common currency.” Instead, people go back and forth, delib¬ 
erating between what is important to them, constructing value hypotheses for compari¬ 
son. In an interesting twist, patients with orbitofrontal cortex damage do not go back 
and forth between options—instead, they look at one option at a time and determine 
if it’s good enough. 68 It is not known, however, whether this change in strategy occurs 
because the orbitofrontal cortex actually controls the strategy used or because the orb¬ 
itofrontal cortex aids in the comparison and the patients recognize that they just aren’t 
very good at doing comparisons. In any case, it is clear that the orbitofrontal cortex is 
critically involved in comparing options during deliberation. 


Action selection 

The final step in deliberation, the taking of the action, is still a nearly completely open 
question. How do people and other animals converge on a decision? Is it a process of sim¬ 
ply saying “is that good enough?” after each evaluation? Is it a process where evaluations 
are held in memory and directly compared? At this point, scientists don’t know, but there 
is some evidence in favor of the comparison hypothesis (that the expected outcome and 
its evaluation are held in working memory and compared with other alternatives). 
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If deliberation entails predicting future options and comparing them, then you have 
to remember each option to be able to compare them. Working memory is the ability 
to hold multiple things in your head at the same time. 69 This would suggest that the 
effect of taxing working memory would be to make people more inconsistent in their 
choices—with fewer working memory resources, there is less time to get the delibera¬ 
tion right, and one becomes more likely to give up the search through futures early. This 
is what the evidence shows. 70 People with better working memory are more willing to 
wait for later rewards. They are also better at making choices. In fact, it is possible to train 
working memory and improve some people’s ability to wait for delayed rewards. 

The advantage of deliberation is that it provides flexibility. Knowing that going to the 
casino will allow me to gamble does not obligate me to go to the casino. Coupling delib¬ 
eration with imagination can allow us to make decisions about very complex decisions 
that we can’t (or shouldn’t) make lots of times (like which job to take or which college 
to go to or whom to marry). Coupling deliberation with attention allows us to push 
ourselves away from choices that we are highly motivated to take (that cigarette) and 
toward choices that are better for us in the long run (exercising at the gym). The prob¬ 
lem with deliberation is that we can get “lost in thought.” Like Hamlet, trapped between 
difficult possibilities, we spend our time predicting outcomes, weighing options, listing 
pros and cons, and never actually taking action. 71 


Tolman and Hull 

I’d like to end this chapter with a short discussion about the history of deliberation and 
its role in decision-making. The difference between deliberative and habit (what we 
have called “procedural”) decision-making became one of the great scientific debates of 
our time. 72 In the 1930s and 1940s, two major laboratories were running rats on mazes, 
trying to determine how they found the goal. (Both of these labs were behaviorists 
in the sense that they wanted to be quantitative and to operationalize the process of 
decision-making in animals that they could study with appropriate controls, in response 
to the introspection of the Gestaltists.) On the East Coast, at Yale University, Clark Hull 
argued that rats learned to associate stimuli with responses, while on the West Coast, 
at Berkeley, Edward Tolman argued that rats were cognitive creatures that mentally 
planned routes to goals. As late as 1970, in a review of this still-ongoing debate, Endel 
Tulving and Stephen A. Madigan wrote that “Place-learning organisms, guided by cog¬ 
nitive maps in their heads, successfully negotiated obstacle courses to food at Berkeley, 
while their response-learning counterparts, propelled by habits and drives, performed 
similar feats at Yale.” 

We’ve already seen how latent learning led Tolman to hypothesize the presence of a 
cognitive map. Tolman also watched his rats learning the maze, and noticed that as they 
made mistakes, there seemed to be a method to those mistakes—rats would make all 
left turns, then all right turns, then they would alternate left and right, as if they were 
actually testing hypotheses. 73 The problem with these observations is that if the rats 
were just making random choices during learning, then sometimes rats making random 
choices would take all lefts, (if there are three left-right choices on the maze, then one 
rat in eight will go left on all three choices even if the rats were really making random 
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choices.) Because our brain likes to find patterns, those rats making all lefts on the three 
left-right choices would stand out. Similarly, even a rat that knows the correct answer 
isn’t necessarily always going to take that choice. (Maybe there’s some noise in the out¬ 
come; maybe the animal is exploring to check its hypotheses.) Actually determining 
when the animal knows the answer is mathematically complicated. 74 

We now have the statistical methods to identify whether these sequences are 
random, 75 but these complex statistics were not available to Tolman. Randy Gallistel, 
Stephen Fairhurst, and Peter Balsam used these new statistical analyses to explicitly 
examine how rats learn and found that (on certain tasks) individual rats didn’t learn 
gradually, but suddenly switched on a single trial from getting it mostly wrong to getting 
it mostly right. 76 They argued that the gradual learning rate seen in many studies was an 
artifact of averaging across different rats, each of whom reached its “Aha!” moment at 
different times. K 

Because it was so hard to analyze the variability in the mistakes made early in learn¬ 
ing, PIull decided to analyze rats who had reached a regularity in their paths and were 
no longer making mistakes. Flull used the analogy of a switchboard operator creating 
a connection between two telephones to explain his ideas, while Tolman had only the 
analogy to human inspiration to attempt his explanation. While Flull was able to pro¬ 
vide a mathematical explanation for his suggestions, Tolman was not. This made it pos¬ 
sible for Flull (and the others arguing against cognitive processes, such as B. F. Skinner) 
to explain specifically when phenomena would occur and made it possible for them to 
be quantitative about it. 78 Science is about prediction and replication. Tolman’s obser¬ 
vations were hard to quantify. Inspiration hit rats at different times. So, while any indi¬ 
vidual rat would suddenly “get it” and start running the maze correctly, the average of 
all the rats showed a slowly developing curve. Similarly, even though each rat would 
individually show vicarious trial and error on some trials, the trials on which the vicari¬ 
ous trial and error behavior occurred would differ from rat to rat. 79 

Mathematics is a technology. Like any other technology, it allows us to see things 
we could not see before. (Compare the microscope, which allowed Robert Hooke to 
see cells, or the telescope, which allowed Galileo Galilei to see the moons of Jupiter. 80 ) 
It is important to realize that Tolman was before Claude Shannon, so he was lacking 
the mathematics of information theory. He was before the first computer (developed 
during World War II), before John von Neumann and Alan Turing, and before Norbert 
Wiener’s breakthrough Cybernetics book (published in 1948), and thus had no access 
to the new conceptualization of intelligence as computation. And he was before Alan 
Newell and Herb Simon, so he was lacking the mathematics of algorithm. 81 Tolman’s 
major work was in the 1930s and 1940s, summarized in his Psychological Review paper 
“Cognitive Maps in Rats and Men” published in 1948. Claude Shannon published his 
breakthrough paper on information theory in 1948. Alan Turing did his major work on 
computability in the late 1930s and through World War II in the early 1940s, but it did 
not have a major influence on psychology until the development of the field of artificial 


K As we saw in our discussion of multiple decision-making systems (Chapter 6) and will return to 
in our discussion of Procedural learning (Chapter 10), there are some tasks that individuals do learn 
gradually and incrementally. 77 Suddenly getting the answer right (an “Aha!” moment) is a sign that a 
task is being solved by the Deliberative system rather than the Procedural system. 
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intelligence in the 1950s. Allen Newell and Herb Simon developed the first algorith¬ 
mic concepts of search and intelligence with their publication of the General Problem 
Solver in 1959, the year Tolman died. 

Information theory quantifies the ability of a signal to be pulled out of noise. 
Algorithm provides the concept that a sequence of computations can be linked together 
through control mechanisms to come to a conclusion that is hidden within data. One of 
the most important ideas, however, was that of computational complexity, which origi¬ 
nated with Turing. We are now familiar with computers running long algorithms—we 
have all seen computers thrashing, the hard drive spinning, computing a complex calcu¬ 
lation. Before the concepts of information theory, algorithm, and computation, the idea 
that results hidden within data could be found only after calculations that inherently 
take time to calculate was unappreciated. 

We now know that both Tolman and Hull were right. When faced with complex 
decisions between similar choices or between changing choices, one needs a flexible 
system even though it may be computationally complex, but when faced with a regular 
decision in which the right answer does not change, animals develop response-chain 
habits (sequences of actions). 82 In practice, early performance on mazes is often accom¬ 
plished by Tolman-like deliberative search processes, while later performance is often 
accomplished by Hull-like stimulus-response habit processes. 83 In fact, if we go back 
and look at their data, Tolman studied how the mazes were learned and concentrated 
his analysis on the early-learning time, while Hull studied “asymptotic performance” 
once behavior had settled down to a regular routine. Both systems (Tolman’s computa¬ 
tionally expensive search process and Hull’s computationally simple stimulus-response 
process) are capable of solving tasks and making decisions. Each system, however, has 
advantages and disadvantages and is accomplished by a different set of interacting brain 
structures. In the next chapter, we’ll turn to what is known about that habit system. 
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The Habits of Our Lives 


Jogging a mountain path, 
one step after another... 

empty the mind 

except for the blue sky above and 
the red rocks beneath our feet 


With enough experience, action-selection becomes "cached" or stored in 
stimulus-response pairs and response-action chains. The advantage 
of this system is that it can respond quickly and reliably, with the same 
action-sequence every time. The disadvantage of this system is that it is 
inflexible and is slow to change in response to changing conditions. 


Poets often refer to us as “creatures of habit.” In the morning, we get up at the same time, 
we drink the same coffee every morning, eat the same cereal. In the evening, we want our 
book or our television before bed. We have all developed our own habits over the course 
of our lives. These habits are not instincts; they are not genetically programmed into us. 
Instead, habits develop out of regularity, reliability. We’d like to think habits develop out 
of what we’ve individually found to be most useful to our lives, but of course, we all have 
habits we’d like to break. Why do we develop bad habits? Where do habits come from? 
What are habits? Why does the route we drive to work become a habit, but the route we 
drive to the airport not? A 

Fundamentally, habits are sequences of actions that we’ve learned to take. The sim¬ 
plest example of a habit is the drive to work. The first time you drive to work, you need 
to use a map, follow a GPS, pay attention to the route you want to take. But if you drive 
the route every day for weeks, months, or years, you eventually stop paying attention to 
it. You drive yourself to work, even as you are thinking of something else. 


A The obvious answer here is that you drive to work a lot more than to the airport (unless you work 
at the airport). As many a business traveler knows, with enough trips to the airport, the route to the 
airport will become a habit too. 
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This suggests the usual trajectory of learning—flexible, deliberative mechanisms 
predominate early in learning, while the inflexible, habit-based mechanisms develop 
with experience. 1 But, of course, control can also shift back from habit to deliberative 
systems. 2 When the 35W bridge that was the primary conduit across the Mississippi 
River through Minneapolis collapsed in 2007, traffic had to be rerouted throughout the 
city. This changed the flow of traffic along my route to work, and I found that the route 
I normally took to work was much slower than it had been. So I found a new route, 
which actually turned out to be faster, even once the new bridge was built and reopened. 
This new route requires turning left at a certain intersection. Yet, even years afterwards, 
if I’m not paying attention, I continue straight at that intersection. Particularly, if I’m 
thinking about something else, like a test I have to write, or a scientific problem, or even 
just trying to remember that I have to pick up milk on my way home after work, I go 
straight through. What’s interesting about this is that no matter how distracted I am 
with my thoughts, I have never run the red light. I have never failed to brake in time and 
I have never hit another car at this intersection, even though it’s a very crowded inter¬ 
section. My brain is driving reliably, but I’m not paying attention to it. 

This does not mean that it’s safe to drive distracted! Driving while talking on a cell 
phone, even a hands-free phone, has been proven to increase the likelihood of crashing 
fourfold. 3 In particular, sensory distractions (such as talking or, worse, texting, on a cell 
phone) interfere with the sensory processing needed to respond appropriately while 
driving. As we will see, there is a big difference between allowing your mind to wander 
to think of additional things and taking your attention off the road. 

This difference can be easily seen in learning to play a musical instrument. No musi¬ 
cian would attempt to play a complex piece while watching television or reading a book: 
the sensory cues would interfere with the music. However, with experience, musicians 
become adept at thinking less and less about details and more and more about the artis¬ 
tic aspects of the music. When learning to play a musical instrument, at first you need 
to think what keys are where, what key combinations produce what notes. Playing a 
musical piece through without playing a wrong note often means pausing before each 
note to find the right fingering (which, of course, completely ruins the rhythm). But as 
one becomes experienced with the instrument, one begins to associate the notes with 
the keys, until at some point you no longer think about the fingering for a given note; 
instead, you think “play a G,” and your hands just do the right thing. This allows you to 
concentrate on the notes. Eventually, as you learn a piece of music, you attend less and 
less to what the next note is, allowing you to spend your mental computation on more 
and more complex aspects of the music. Expert musicians with lots of experience with 
different pieces and different genres can even create complex musical melodies on the 
spot (see jazz, improvisation). 

Computationally, habit learning arises from two important processes: situation- 
action B (S-A) associations and chunking (action-action, A- A) processes. S-A associations 


B In the psychology literatures, “situation-action” learning is more often referred to as stimulus- 
response (S-R) learning, 4 but, as we’ve seen (Chapter 6), animals are reacting to situations, catego¬ 
rizations of stimuli, not to specific stimuli. 5 (We’ll discuss this issue in more depth in Chapter 12.) 
Certainly some stimuli are more important than others in determining the situation at hand, but rarely 
is a situation defined entirely by a single stimulus (outside of a psychology animal-testing box where 

( continued ) 
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are just what they sound like, memories that have learned that in this situation, you should 
take this action. 6 Chunking is the chaining of A-A sequences, as if a sequence of actions 
was represented as a single “super-action.” 7 (The habit system is sometimes called the 
“cached-action” system, because it has cached or stored the action to take. 8 In Chapter 6, 
we called this system “Procedural,” because it learns and stores procedures. 9 ) 

It is interesting to note that most of the examples of the “stimulus-reaction” type of 
behaviors typically identified as “habits” are actually voluntary initiations rather than 
responses—when the light turns on, the rat pushes the lever; when given the cigarette, 
the addict smokes it. Although the drive to work becomes an automatic “habit” and no 
longer requires attention through the journey, it’s still a voluntary action initiated and 
monitored throughout. The system is retrieving a voluntary sequence of motor actions 
and releasing it in response to a cue. 10 

This is exactly how athletes play sports. A quarterback throwing a pass to a receiver 
is recognizing the situation, evaluating it, and retrieving an action sequence. No matter 
what some fans will say, no one “made” the quarterback make the throw that led to the 
interception; he placed “throwing the ball” on the set of actions he was ready to do and 
then misread the coverage. Procedural decision-making is best understood in terms of 
preparation and possibilities —at each moment, the agent is matching situations until one 
appears that is matched with a releasable action. 11 ' 0 

Chunking is a process where subgoals are stored as completed parts so that the cogni¬ 
tive planning process can work at a more abstract level. 13 Thus, the expert musician does 
not think “to play a G on this clarinet, I need to put my fingers in this position and blow 
this hard on the reed”; instead, the musician just “plays G.” In fact, expert musicians often 
chunk sequences of notes in chords and arpeggios. Similarly, a baseball pitcher doesn’t 
think “to throw the fastball, I place two fingers on the seams of the baseball and pull my 
arm back, don’t forget to turn the hips....” Instead, the expert pitcher just throws the 
baseball. As the catcher Crash Davis (Kevin Costner) says to the pitcher Ebby LaLoosh 
(Tim Robbins) in the movie Bull Durham, “Don’t think, meat, just throw.” 14 This advice 
is key because thinking too much about the process disrupts both the S-A component 
(because it forces the deliberation component to take over) and the chunking compo¬ 
nent (because it is working at too low a level). As we saw in Chapter 9, deliberation takes 
a variable amount of time to get to the next action. A reliable, smooth, action sequence 
needs the Procedural system. 

In the laboratory, these issues are studied with tasks like the sequential serial reaction 
time (SSRT) task. In this task, the subject’s hand is placed on a keyboard and the subject 
is simply told to push each key as soon as the letter appears on the screen. 15 The sub¬ 
ject is told that the letters will appear randomly; however, unbeknownst to the subject, 
the letters actually come up in a sequence. The sequence can be simple or complicated, 


only a couple of very discrete stimuli are available). Therefore, I prefer the term “situation” to “stimulus” 
(conveniently, both start with S). I don’t have any problem with the term “response” per se, but I keep 
getting the R in S-R confused with “reward,” so I prefer the less ambiguous “action,” allowing us to call 
this S-A learning. 

c In the psychological literature, this has historically been known as affordances, because each situa¬ 
tion “affords” (provides for, encourages) certain responses more than others. 12 



90 


THE DECISION-MAKING SYSTEM 


deterministic or probabilistic. The key is that the sequence is never explicitly told to 
the subject. As people learn to recognize the sequence, they begin to make anticipatory 
movements and their reaction time speeds up. This can be explicitly tested by putting in 
a random sequence probe session in which the sequence of letters reverts back to being 
random. The anticipatory movements are no longer useful; in fact, one has to undo the 
preparation, and ones reaction time slows down dramatically. 

What’s particularly interesting about the SSRT task is that there are two potential 
things one can learn about the sequence: one can learn that there was a sequence and 
one can learn how to respond to the sequence. To test whether the subject recognized 
that there was a sequence, we simply ask. Afterwards, when asked, some people will say, 
“Yes, there was a sequence” and others will say, “No, it was random.” To test whether the 
subject learned to respond to the sequence, we measure the reaction times. If the subject 
shows anticipatory movements and speeds up for repeated sequences (and slows down 
in response to the probe session), then the subjects actions in some sense “know” the 
sequence (because the actions are predicting it). 

We might think that these two tests should always match—if you recognize the 
sequence, your reaction time will speed up and if your reaction time speeds up, then you 
must have recognized the sequence. But this is not what is seen. Although many peo¬ 
ple do, in fact, both report the sequence afterwards and show reaction time speed-up 
during the task, many subjects do not recognize the sequence explicitly but still show 
strong reaction-time effects. 16 There are even subjects who recognize the sequence but 
do not show reaction-time effects. 17 In particular, patients with Parkinsons disease and 
Huntington’s disease, both of which affect the basal ganglia, often recognize the sequence 
without showing reaction time changes. In contrast, patients with Alzheimer’s disease, 
which affects learning abilities in the hippocampus and its related areas, often show nor¬ 
mal reaction time speed-up but an inability to report recognition of the sequence. This 
distinction is often referred to as implicit versus explicit learning, which corresponds 
closely to the distinctions between procedural and declarative, between habit and 
goal-oriented decision-making, and the terms used in this book (between procedural 
and deliberative decision-making), all of which seem to be capturing this fundamental 
distinction between an unconscious revealed learning and a conscious, explicit learning 
process. 18 

The two brain structures most involved in both chunking and S-A associations are 
the basal ganglia, 19 sitting at the center of the mammalian brain (beneath the cortex, thus 
“subcortical”), and the cerebellum, 10 at the base of the brain, both of which are evolution¬ 
ary very old structures, with similar components being found in all vertebrates. 21 


The basal ganglia 

The circuitry in the basal ganglia is very complex. We will spend some time exploring it 
in this chapter, but briefly, there are two pathways through the basal ganglia: one encour¬ 
ages actions and the other inhibits actions (Figure 10.1). 22 (Actually, there are three 
pathways through the basal ganglia; the third one stops a learned action. 23 We will return 
to this third pathway in Chapter 15 in our discussion of self-control and the overrid¬ 
ing of learned actions.) The two primary pathways can be identified by their influences 
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Figure 10.1 Pathways Through the Basal Ganglia. The basal ganglia are a 
subcortical circuit that learns to control motor actions. There are three pathways through 
it. (1) The first is a “direct path” in which the cortex excites the striatum, which inhibits 
the internal globus pallidus (GPi). Because GPi neurons are tonically firing, inhibiting 
them releases thalamocortical and subcortical motor action circuits. (2) An “indirect path” 
adds an intervening inhibitory step (external globus pallidus, GPe), which creates a triple 
negative. (3) The third is a “hyperdirect pathway,” in which the excitatory subthalamic 
nucleus (STN) increases firing in the GPi, which stops actions. Learning occurs in the 
striatum through dopamine inputs coming from the ventral tegmental area (VTA) and the 
substantia nigra pars compacta (SNc). 


on thalamic and cortical action-selection systems. The “go” pathway, which encourages 
actions, inhibits cells that inhibit thalamic and cortical action-selection systems (a dou¬ 
ble negative, which encourages actions), while the “no-go” pathway, which discourages 
actions, adds a third inhibition step (a triple negative, which discourages actions). The 
third pathway steps in at the last stage, directly exciting those cells that inhibit the thal¬ 
amic and cortical action-selection systems (a single negative, discouraging actions). 

The first pathway, which encourages actions, recognizes increases in dopamine, 
while the second pathway, which discourages actions, recognizes decreases in dop¬ 
amine. 24 (Remember that dopamine is a signal where increases mean “that was better 
than I expected” and should lead to an increase in the willingness to take an action, 
and decreases mean “that was worse than I expected” and should lead to a decrease 
in the willingness to take an action. See Chapter 4.) These changes in dopamine are 
recognized by different receptors (prosaically called D1 and D2 receptors). This means 
that slight differences in genetics can make one person learn more easily from positive 
rewards (carrots) and another learn more easily from negative punishments (sticks). 25 

This difference suggests that genetic changes in these receptors might change the abil¬ 
ity to learn from positive or negative information. In a remarkable experiment, Michael 
Frank and his colleagues found exactly that. 26 They developed a task in which people 
learned to recognize which arbitrary characters 13 were likely to lead to reward and which 


D Actually, they used Japanese hiragana characters. Presumably, their American experimental sub¬ 
jects didn’t recognize them. 







92 


THE DECISION-MAKING SYSTEM 


were not likely to. Through an elegant combination of training and test procedures, they 
were able to separate learning from positive information (more likely to lead to reward) 
than from negative (less likely to lead to reward). They did genetic testing on their 
subjects and examined one common genetic polymorphism® in the D1 receptors and 
another in the D2 receptors. The polymorphism in the D1 receptor pathway affected 
how well people learned from positive signals, while the polymorphism in the D2 recep¬ 
tor pathway affected how well people learned from the negative signals.® 

As we also talked about in Chapter 4, dopamine is primarily about positive rewards 
(reinforcement), not negative punishments (aversion). We argued in that chapter that 
there should be a negative counterpart to dopamine that signals when punishments 
are worse than expected (learn not to go), and when punishments are not as bad as 
expected (relief, learn to go). As we noted in that chapter, it is not currently known what 
that signal is. In the past few decades, people have suggested two possible candidates for 
this negative anti-dopamine: serotonin and norepinephrine (also called noradrenalin). 28 
Both of these neurotransmitters are chemically related to dopamine (and thus are good 
candidates from the perspective that evolution tends to work by modifying existing 
systems) and are definitely involved in decision-making. 

Dopamine, serotonin, norepinephrine (noradrenaline), and another related neu¬ 
rotransmitter, octopamine, are all related chemicals from a family called the cate¬ 
cholamines. 29 Catecholamines play interesting roles in learning and decision-making in 
a number of very distantly related species. In insects, the positive (reinforcement) com¬ 
ponent is octopamine, while the negative (aversion) signal is dopamine. 30 Dopamine, 
serotonin, and octopamine also control the activation and stopping of movement (swim¬ 
ming, crawling) in simple animals such as leeches, worms, and lampreys. 31 This suggests 
that the role of these catecholamines in encouraging and discouraging actions has a 
long evolutionary history. How the roles that these catecholamines play in these other 
animals relates to their role in human decision-making is, however, still unknown. 

Recent experiments have found that serotonin relates to the ability to hold mem¬ 
ories across delays and to one’s long-term perspective, but that serotonin does not 
reflect negative value-prediction error signals. 32 In contrast, norepinephrine is involved 
in negative affect, particularly in drug addiction, such as during opiate withdrawal. 33 
Norepinephrine bursts drive pauses in the firing of dopamine neurons, which we’ve 
seen reflect disappointment and recognition of undelivered rewards. 34 However, other 
researchers have proposed that norepinephrine plays roles in attention and in repre¬ 
sentations of uncertainty. 35 (Dopamine also plays a role in vigilance and attention. 36 ) 


E Any given species has a distribution of subtle genetic changes between its members—that’s why 
we’re not all identical. It’s these small differences that are the key to evolution, which changes the dis¬ 
tributions of these small differences over time. 27 These differences are called polymorphisms—poly 
(multiple) + morphisms (forms). 

F They also tested a third polymorphism in a dopamine receptor that affects prefrontal cortex. This 
third polymorphism affected people’s ability to respond to changes in the reward contingency. As we 
will see later in Chapter 12, the prefrontal cortex is important for the situation-recognition box, par¬ 
ticularly in the face of changes, and these dopamine signals are used to determine when one needs to 
change one’s mind about the situation one is in. 
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As with dopamine, it is likely that both serotonin and norepinephrine carry general 
signals that can be used in many ways. G 

Whether the basal ganglia are actually driving movement itself or whether the basal 
ganglia are merely influencing decisions being made in the cortex is a subject of con¬ 
troversy in neuroscience today. What is most likely, of course, is that both theories are 
correct and that the action suggestions made by the basal ganglia are being sent to both 
deep brain motor structures (and then on to the spinal cord) and to cortical circuits 
(which then project back to those same deep brain motor structures and spinal cord). 40 


Motor cortex 

How can cortical circuits make decisions? Motor control cortical structures are catego¬ 
rizing inputs from sensory structures. 41 Sensory structures are categorizing aspects of 
the world, with those aspects becoming more and more complex as they move away 
from the primary sensory areas. 42 (in the primary visual cortex, the “categories” are 
really “features of the world,” dots, lines, blobs of color, corners, etc. In secondary and 
tertiary visual cortices, other “features” are found until they become categories, such 
as faces, houses, objects, etc. Similarly, the primary auditory cortex recognizes features 
of sound, and secondary and tertiary auditory areas recognize speech, words, music, 
genres of music, etc.) The output of a category of situations can be treated as an action 
signal. 43 In a sense, this can be thought of as releasing actions—the quarterback cocks 
his arm back to throw, and starts checking receivers. If he finds one that is open (that 
his categorization process identifies as a good throw to make), the motor cortex sends 
a signal to other structures (including the basal ganglia, cerebellum, and deep motor 
structures) to initiate the action of throwing the ball. Notice how this mechanism main¬ 
tains the decision of whether to initiate the voluntary action within the cortical catego¬ 
rization system, but the action sequence itself is not part of that system. The quarterback 
has chunked the decision—the motor cortex determines “throw or not?” but does not 
identify the individual motor motions to take. 

Computational models of the cortex suggest that the categorization process depends 
on a network of cells “settling” on an answer. 44:11 Initiating a decision too quickly in these 
models will pull up the wrong answer. 45 Studies of cortical responses to motor selection 


G This issue is further complicated by the fact that these neurotransmitters have effects in the brain 
beyond value-prediction error, 37 and that all of these neurotransmitters also have effects outside the 
brain. 38 Serotonin has effects on the gut and intestines. Norepinephrine has effects on heart rate and 
blood pressure. Dopamine and norepinephrine affect the peripheral sympathetic and parasympathetic 
systems of the body (unrelated to the emotion of sympathy, of course). The sympathetic and parasym¬ 
pathetic systems run counter to each other: the sympathetic system prepares the body for stressful 
activity (fight, flight, and the act of reproduction), while the parasympathetic system maintains the 
body at unstressful activity (safety, rest, and postcoital relaxation). 35 The complex diverse effects of a 
given chemical on different structures mean that drugs often have unintended side effects and make 
pharmacology complicated. 

H The computational mechanisms of this settling process (basins of attraction in content-addressable 
memories ) are well understood. A detailed discussion of this mechanism can be found in Appendix C. 
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find that there are two responses to sensory cues in motor cortex, a very fast (less than 
50 milliseconds) response and then a slower (greater than 100 milliseconds) response. 46 
The first response occurs through direct connections between sensory cortices and the 
motor cortices; the second response occurs through a longer path through more frontal 
circuits. For example, when asked to turn away from a cue, motor cortical neurons ini¬ 
tially represent the direction to the cue, but then, influenced by information processed in 
premotor frontal circuits, change their firing to reflect the direction of action. 47 

This issue can be best seen in terms of reaction to a skid on ice. The first reaction 
most people have is to brake and turn against the skid, but this only makes things worse. 
In fact, the correct thing to do in a skid is to slow down by releasing the accelerator, and 
then to turn into the skid until you get traction back. 48 

This poses an interesting problem with brain-machine interfaces. Many people 
have argued that brain-machine interfaces could speed up reaction times. But part of 
the slowness of reaction time is that your brain has to develop the correct response. 
Decoding the information from the motor cortex too quickly can produce the wrong 
answer. As with driving during a skid, you don’t want the first reaction, you want the 
right reaction, which sometimes appears second, only after the brain has had a chance to 
do some additional processing to find the right categorization of the complex situation. 

It is important also to remember that these categories are actually dynamic— 
although it is easier to build computational models of categories as a static value, 
the cortex is actually finding a path through these categories. 49 This path is actually a 
sequence of categories—my drive to work is to first go down Roselawn, then left on 
Hamline, and right on Larpenteur... At each point, I need to correctly execute behavior 
at two different levels—at one level, I need to execute the actions correctly, stop at the 
traffic light when red, go when green, don’t hit that car in front of me, etc. And at the 
other (usually termed “higher” level because it’s more abstract), I need to decide on 
the next step in our journey—which way do I turn at this intersection? The point of 
this chapter is that sometimes even those higher-level decisions can be unconscious. 
However, even when we are consciously deciding which way to turn at an intersection, 
we are not usually consciously thinking “red light means stop” or “green light means 
go.” This is chunking 50 —we have stored the behavior in a chunk so that we can think 
“turn left,” not “take your foot off the brake, put your foot on the accelerator, turn the 
wheel,” etc. A computer programmer might say that we have compiled the process into 
a subroutine that we can access easily. Of course, the cognitive brain can still break into 
these chunks and control the components—as when we are trying to teach someone 
to drive and say, “It’s a red light! Stop!” as we slam our foot into the imaginary pedal on 
the passenger’s side of the car. 

In general, driving is a great example of this chunking process. I remember my first expe¬ 
riences behind a wheel of a car, trying to figure out how much pressure I dared put on the 
accelerator, as the car suddenly lurched forward. Or the nervous way I clutched the steering 
wheel the first time I got on a highway with cars whizzing by at 70 miles per hour. Other 
great examples of this sort of chunking are playing a musical instrument and walking. 

Walking on two legs is actually a complex interaction of balance, forward motion, 
and catching yourself. As Laurie Anderson sang in her 1982 song Walking and Falling, 
“You’re walking. And you don’t always realize it, but you’re always falling. With each 
step you fall forward slightly. And then catch yourself from falling. Over and over, you’re 
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falling. And then catching yourself from falling. And this is how you can be walking and 
falling at the same time..Anyone watching a toddler take its first steps can see how 
complex a process it is. 


Cerebellum 

The key to these compiled behaviors is the cerebellum, a fascinating brain structure 
that sits at the back of the brain, hanging out over the spinal cord. In a human brain, 
the cerebellum looks like an addendum, an extra appendix bulging out from the back 
of the brain, but in fact the cerebellum is an evolutionarily much older structure than 
the cortex. The cerebellum first appears evolutionarily with vertebrates, and it has not 
changed in basic structure from early fish to mammals. 51 (Because the cerebellum is a 
distinctive bulbous structure, the skulls of many animals have pockets to accommodate 
it. Thus, the location of the cerebellum can be seen in the fossils of ancient animal skulls, 
including fish, and even dinosaurs. 52 ) 

The structure of the cerebellum is fascinatingly simple and regular. 53 The main 
component are the Purkinje cells, 1 which have dendritic trees that are remarkably 
two-dimensional, spread out in sheets like pieces of paper. They stack against each other 
like pages in a book. Two inputs converge on the Purkinje cells: the parallel fibers and 
the climbing fibers. Interestingly, like the basal ganglia outputs, the Purkinje cells are 
inhibitory. Purkinje cell firing holds down its outputs until releasing that output by a 
sudden decrease in firing. 

The cerebellum looks like a great big spreadsheet, with the parallel fibers and the 
Purkinje cells lined up perpendicularly. The other major input to the Purkinje cells are 
the climbing fibers, which climb the Purkinje cells like ivy on a tree, and provide a teach¬ 
ing signal. 54 Errors in the smoothness of the actions are transmitted to the Purkinje cells 
and the connections between the parallel fibers and the Purkinje cells are adjusted until 
the error is diminished. 55 

A classic example of this learning signal is the prism experiment, 56 shown to many a 
high school student as part of neuroscience outreach programs. In this experiment, the 
subject is given a set of prisms that shift the visual world to one side (say 20 degrees to 
the left). The person tries to throw a beanbag into a basket a few feet in front of him or 
her. Because there is a mismatch between the visual world and the motor actions the 
person is used to, the person throws the beanbag 20 degrees to the opposite side (to the 
right). Over the course of a dozen throws or so, the subject adjusts his or her behavior 
to correctly toss the beanbag into the bucket. Then, when the prisms are removed, there 
is again a mismatch and the person throws the beanbag 20 degrees to the original side 
(to the left). Again, over the course of a few throws (fewer than before), the subject 
recaptures the old association and is able to hit the basket again. These changes occur 
due to connectivity changes in the cerebellum. 

At a neuroscience outreach fair to a local high school, we once encountered a person 
who showed no effect of the prisms on throwing the beanbag (although he had never 


1 Pronounced “PURR-kin-jay” after Jan Purkinje (1787-1869), a Czech anatomist from the 1800s, 
who first noticed the beautiful fan-like structure of the Purkinje cells. 
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worn prisms before); he hit the basket on the first try, every try. At first we were very sur¬ 
prised, but we later learned that he was a local basketball star. Presumably, he had taught 
his motor system to throw straight independently of what his vision told him. He had 
already identified the distance to the basket before he put the prisms on. He knew he 
was facing forward and didn’t have to use the visual signals to correct it. He had already 
chunked how to throw a certain distance and angle. Through extensive practice, he had 
developed a compiled motor action that reliably hit the basket. 

The cerebellum seems to be involved in these compiled or chunked motor actions— 
walking, driving, playing a musical instrument. The cerebellum is the key to timing reliable 
motor actions and sequences. 57 There is some evidence that the cerebellum can also com¬ 
pile cognitive sequences as well, but this is still controversial. 58 (it’s much harder to mea¬ 
sure a cognitive sequence than a motor sequence.) However, that the cerebellum has a role 
in cognitive sequences shouldn’t surprise us. People can certainly chunk mental cognitive 
sequences just like they do physical motor sequences. 59 (What'syour cellphone number ?) 


Summary 

In this chapter, we’ve seen two forms of arbitrary action decisions—the caching of 
stored decisions at a high (abstract) level and the compiling of stored motor sequences 
at a low (concrete) level. The former uses the basal ganglia to identify the correct next 
step and the latter uses the cerebellum to time the motor action sequences. But both of 
these motor control/decision-making systems can be contrasted to the three other sys¬ 
tems we’ve been discussing—reflexes are hardwired (evolutionarily) into our system, 
they cannot change either what sensory stimuli trigger them or what actions they take; 
Pavlovian/emotional systems cannot change the actions triggered but can learn to asso¬ 
ciate new stimuli with them; Deliberative systems can decide to take arbitrary actions 
in response to arbitrary situations but require extensive attention and computational 
time to process. Compiling action sequences allows us to make decisions about chunks 
of behavior rather than worrying about every little detail. And caching situation- 
action pairs allows us to attend to other plans while taking even complex arbitrary 
actions in response to complex stimuli. In the next few chapters, we will discuss the addi¬ 
tional systems needed to understand the human decision-making machine (perceptual, 
situation-recognition, and motivation systems), and then we will move on to discuss 
issues of interaction between the systems (the tradeoffbetween exploration and exploi¬ 
tation, and the process of self-control), before discussing the consequences of the physi¬ 
cal brain and its potential vulnerabilities and failure modes. 

Books and papers for further reading 
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Integrating Information 


Leaves in the wind 
or a leopard in the forest? 
Don’t take too long to decide! 


Although our senses receive physical information from the world, 
we interpret those senses in terms of the objects in the world—we 
see a tree, we feel the wind, we hear a bird. Interpreting that infor¬ 
mation requires an integration process that takes time. We can 
trade off time for accuracy if need be. 


When you look at a picture (say of a face), your eyes dart from one important feature 
to the next. 1 In part, you need to do this because the photoreceptors in the human eye 
are more concentrated in the center of your retina than in the periphery, which means 
that your vision is more accurate (has higher resolution 11 ) when you are looking directly 
at something. Every few hundred milliseconds, your eyes make another journey (they 
saccade B ) to another location. Over the course of a few seconds or minutes, you have 
integrated the important information about the picture to answer whatever question 
you had. Where you look, how long you look, and how long your eyes stop at each point 
depend on the questions you are trying to answer about the picture. 0 

Experimental studies of decision-making often make the perceptual recognition of the 
information difficult, which means that one has to integrate information over time. 2 Old 
studies did this because it allowed them to use the reaction time to differentiate between 
conditions. New studies do this because it allows them to study how the neural firing 


A We are all familiar now with issues of visual resolution thanks to the digital camera industry. 

B We are rarely aware of these saccades, although, like most motor systems, they can be taken under 
conscious control. 

c It is possible to identify information about the world without looking directly at it. (As you might 
if you were trying not to let someone know you were looking at them. Because human eyes have white 
sclera, we can tell where someone else is looking, an important issue for social creatures.) Not looking 
at something takes more effort because one has to build up the information from the sensors at the 
periphery with lower resolution, and because one has to inhibit the natural tendency to saccade to it. 
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changes as the subject figures out what to do. Interestingly, because of this, much of the 
neuroscience literature on “decision-making” is about situation recognition rather than 
deliberation or the other action-selection processes we have been discussing. In these per¬ 
ceptually difficult tasks (which we discuss in detail later), the hard part is recognizing which 
situation the cues imply you are in not deciding what to do. Each situation has a single cor¬ 
rect outcome. (Compare this to the examples we discussed in Chapter 9 when we talked 
about people deliberating over job options or over large purchases like cars or houses.) 
Nevertheless, perceptual categorization is a key part of the decision-making process, and 
these perceptually difficult tasks have taught us how we integrate information over time. 


Race-to-threshold models 

Computationally, integrating information is most easily understood in terms of the 
race-to-threshold models. 3 In these algorithms, originally used by Alan Turing and the 
Allies to break the German Enigma code in World War II, D the weight of evidence for 
each possibility is measured, and when the weight of evidence becomes overwhelm¬ 
ing for one possibility over the others (when the weight of evidence for one possibility 
reaches a threshold), that choice wins and the decision is made. 

Let us first look at this in the case of a two-choice forced decision. A two-choice 
forced decision is one where you must make a decision between two options. There is 
no third option; not choosing the first option is equivalent to choosing the second. We 
can measure how much better one option is than the other by measuring the ratio of 
the weight of evidence for one option over the weight of evidence for the other option. 
The ratio of the probabilities is convenient to use because any advantage to one side is 
equivalent to a disadvantage to the other side. 

Some readers may recognize this as an odds ratio, 4 which is usually written as 1:X, 
where X measures how much better the second choice is than the first. However, these 
odds ratios require a lot of division and require that we represent our number from 0 
(ldnfinity) to infinity (infinity: l). Abetter option is to take the logarithm of this ratio. 
The advantage of logarithms is that multiplying two numbers is equivalent to adding 


D Alan Turing (1912-1954) could arguably be said to have saved the world. Turing led the 
code-breaking team in Bletchley Park in England that broke the Nazi codes. The Germans had devel¬ 
oped a code that was so complex they thought it was unbreakable. It was based on a machine, called 
Enigma, which consisted of three dials. The person typed a message into the machine and the machine 
encoded the message on a character-by-character basis as a function of the dials. The dials were set in a 
specific way each day and so the code-breaking team had to break each day’s code anew. But a captured 
machine gave the Allies the set of possible dials. Even if they didn’t know the settings for that day, 
they knew the parameters and could calculate the likelihood that a given message came from a given 
dial setting. Because thousands of messages were sent each day, Turing and his team could integrate 
the information between these messages. They measured the weight of evidence for each setting and 
stopped when they reached a threshold. Although they didn’t crack the code every day, they cracked it 
often enough that they could read the German orders and they knew what the Germans were going to 
do. The Germans thought the Enigma was impenetrable and continued to send messages by it until the 
end of the war. Both David Kahn’s nonfiction history book The Code-Breakers and Neal Stephenson’s 
fictional novel Cryptonomicon tell this story in wonderfully exciting and readable detail. 
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their logarithms; similarly, dividing two numbers is equivalent to subtracting their loga¬ 
rithms. The logarithm is a function that translates this ratio into a number from minus 
infinity to plus infinity. (This turns out to be how much information there is to sup¬ 
port one hypothesis over the other.) When the two choices are equal, this difference 
between the logarithms is zero, and the difference moves away from zero as we gather 
information. This allows us to define the weight of evidence of choosing left or right as 
a single number (greater than zero will mean go left, less than zero will mean go right). 
We simply start at zero and integrate our information over time by moving up with 
evidence for the first choice and down with evidence for the alternate choice. 

This leaves us with three important parameters. First, we have the slope, which is how 
much to change our ratio with each piece of information supporting one hypothesis over 
the other. Second, the bias determines where we start our decision ff om. E Finally, third, there 
is the stopping condition, which can either be taking the best option after a certain time or 
(more commonly used in the experimental literature) the odds ratio where we decide that 
we have enough information—that is, when we have deviated from zero (the two choices 
being equal) to some threshold (one choice is so much better, we ll go with that one). 

I find the race-to-threshold model extremely intuitive in that it entails weighing the 
pros and cons of two options and stopping when we’re convinced that one choice is better 
than the other one. The primary parameters are ones that make sense intuitively— bias 
is our preference for one choice over the other, slope is how quickly we learn about the 
options, and the threshold is how sure we want to be before we commit to a choice. 

The integration-to-threshold model produces a unique distribution of reaction times 
because of the way the integration occurs (Figure ll.l). 5 It works out to be related to 
what is known in the engineering fields as a hazard function— How long will it take before 
something happens?, given that we stop counting the first time it happens. This is used, 
for example, in measuring the first time something breaks down and has to be repaired 
(like an airplane). 6 It also tracks the physics of a noisy signal crossing an energy barrier. 7 
Imagine popcorn in a hot-air popcorn popper. We have two cases of a “hazard” function 
here. First, the popcorn is bouncing around, and some of the popcorn will have enough 
bounce to get out and over the threshold of the popper on its own. The more the air 
blows, the more the popcorn bounces (more air = faster slope; getting out of the pop¬ 
per = crossing the threshold). Second, the heat in the popper is making the water inside 
each kernel of popcorn expand; at some point the water expands enough and blows the 
kernel open (more heat = faster slope; exploding kernel = crossing the threshold). If the 
first hazard function occurs before the second, you get unpopped kernels in your bowl. 

These theories have been explicitly tested in a number of experimental paradigms. 
The simplest is the “visual noise” or “television snow” paradigm/ In this paradigm, the 
subject stares at a black screen on which bright dots are plotted. Some of those dots 


E Interestingly, the English word “bias” has two related meanings. It can mean the initial expec¬ 
tation, an assumption made before gathering information. Second, it can mean a preference for one 
thing over another, particularly when one responds without appropriately integrating the information. 
As pointed out in The Anatomy of Bias by Jan Lauwereyns, these are really the same thing—a stereotype 
is an initial expectation (often wrong) that is unwilling to change in the face of further information. 

F Most televisions (and all computer monitors that show television signals) are digital now, but 
older televisions received analog signals. Analog televisions that were not tuned to a signal would 

( continued ) 
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Figure 11.1 Integration to Threshold. (Left) Multiple accumulators start with 
an initial bias, and each races toward threshold by integrating the accumulating evidence 
at a different rate. The first one to reach threshold wins, and the perception collapses to 
decision. (Right) Individual neurons in LIP integrate information. All trials start at similar 
firing rates (there is no initial bias) and end at the same firing rates (on reaching threshold) 
but integrate at different rates depending on the coherence of the incoming sensory signal. 
Right panel from Gold and Shadlen (2002), redrawn from Roitman and Shadlen (2002). 
Reprinted with permission of author and publisher. 


vanish in the next instant. Other dots move in a random direction. But some of the dots 
move together in a given direction (say left). The goal is to figure out which direction the 
subset of coherent dots is going. By varying the proportion of dots moving in the same 
direction, the experimenter can quantitatively control the speed at which information 
is provided to the subject. Both humans and monkeys are remarkably sensitive to even 
very small proportions, being able to easily recognize as few as 10% of the dots moving 
in the same direction. 8 

This task depends on the primate ability to detect motion. Your brain wants to detect 
things that move for two reasons. First, if you are moving, then the whole world you 
see moves in sync. (This is the secret to the illusions of motion that IMAX movies can 
produce.) But, second, your brain wants to detect other things that move in the world. 
The ability to detect things that move depends on a brain structure called MT, G which 
has cells that detect things that move in the visual world, and a structure called LIP, H 
which has cells that have tuning curves to a given direction and increase their firing rate 


show a random mixture of black and white dots that was called “snow.” The visual motion signals being 
shown in the visual noise paradigm look a lot like this analog television noise. 

G The structure is now known solely by its acronym “MT,” but at one time this acronym stood for 
medial temporal cortex (meaning “in the middle” [medial] of the temporal cortex). The temporal cortex 
is located underneath the rest of the brain. In the early literature, MT was also known as V5, since it was 
the fifth visual cortical area discovered. 9 

H Again, now known by its acronym (LIP), but referring to lateral intraparietal cortex, because it sits 
on the internal wall of one of the grooves in the parietal cortex. 10 
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over time as the monkey watches the screen. (There is a nearby structure called MST/ 
which detects the first case, where the whole visual world is moving, usually because 
you are moving within it.) The features of the world detected by cells in MT are good 
for detecting moving animals and objects. Imagine, for example, a camouflaged animal 
(like a leopard) moving through a forest. Most of the dots you see would flicker on and 
off with the wind, but a few of them (the leopard!) would reliably move in the same 
direction together. 

The MT cells provide information about the direction of those moving dots—each 
cell’s firing rate has a preferred direction and the firing of that cell reflects the proportion 
of dots moving in that direction against a static background. 11 (MST cells also have a 
preferred firing rate and reflect motion of the entire screen. 12 ) 

The LIP cells seem to be integrating the information. (MT projects directly to LIP.) 
The starting firing rate of an LIP cell reflects the monkey’s initial predictions about 
whether the dots are going to be moving in that cell’s favorite direction or not (the bias), 
the speed at which the firing increases reflects the information coming from MT (the 
slope), and when the LIP cells reach a certain firing rate, the animal reacts (reaching 
the threshold). 13 We can think of this decision as being controlled by two neurons, one 
which is looking for leftward motion and one which is looking for rightward motion— 
the first to reach threshold wins, and that’s the decision made. 

Now that we’ve seen how this integration-to-threshold idea can work with two 
choices, it is straightforward to imagine how it would work with more choices. 14 For 
example, I doubt that monkeys evolved to make a choice between whether dots were 
going left or right. More likely, these experiments are tapping into a system that evolved 
to detect the direction (any direction) that a camouflaged object was moving. Instead of 
two neurons, we have hundreds or thousands of neurons, representing a range of pos¬ 
sibilities, each integrating the information, each racing to threshold. 

This is something called a winner-take-all network and arises from a simple model 
combining excitatory and inhibitory cells (cells that increase the firing of the cells 
they send axons to and cells that decrease the firing of the cells they send axons to). 
The excitatory cells are assumed to be connected up such that cells that represent 
similar things (say similar directions) support each other, while cells that represent 
different things (say directions 180 degrees apart) don’t. 15 The inhibitory cells work as 
simple negative feedback, so that the total activity in the excitatory cells must remain 
a constant. This network was first examined in the 1970s as a model of cortex, and it 
turns out to be a very reasonable model of many aspects of the cortex. Mathematical 
work in the 1980s showed that the stable final state of this network is an agreement 
between the excitatory cells, with a subset of cells that agree on a certain result all 
having high firing rates and cells that disagree with that result having low firing rates. 
This network was used in the 1990s as a model of the rat’s sense of direction and 
the rat’s sense of location (place cells in the hippocampus) and as a model of cor¬ 
tex. Functionally, these models are a special case of the content-addressable memories 
discussed in Appendix C. 


1 Again, now known entirely by its acronym (MST), but once meaning medial superior temporal 
cortex, “superior” meaning “above.” 



102 


THE DECISION-MAKING SYSTEM 


Parallel and serial perception 


A radar sweeping around the sky, looking for incoming airplanes, is a serial perception— 
it can only see one direction at a time; each direction is checked sequentially An ana¬ 
log camera perceives light in parallel, with each light-detecting chemical on the photo¬ 
graphic plate reacting at the same instant. (Digital cameras use a combination of parallel 
and serial processing. Each charge-coupled device on each pixel of the camera detects 
light at the same time [integrating the photons over the time the shutter is open], but 
then the pixels are usually read into the memory serially.) Our perceptual systems use a 
combination of parallel and serial processing, but the parallel step generally goes much 
farther than simple light detectors. 16 

It’s useful to examine this parallel and serial processing in vision. Light is detected 
by the retina, which works in a massively parallel manner, like the pixels on our digital 
camera, integrating the photons that reach our eyes. From there, signals are sent on to 
the thalamus, and then to the primary visual cortex, which contains feature detectors, 
particularly for dots, lines, and edges. Each of these feature detectors is repeated for each 
part of the visual field, as if for the pixels of our eyes. From there, signals are sent to many 
different visual systems, which attempt to pull information out of the signal in paral¬ 
lel. One system, for example, tries to identify objects, another tries to identify faces,- 1 
another recognizes objects moving through the visual field (MT). 

The classic experiment examining the difference between parallel and serial percep¬ 
tion is the pop-out experiment (Figure 11.2). 20 Your goal in this task is to identify a 
certain object (the target) in a field of distractors. When the target is really different 
from the distractors (O in a field of Xs), the target seems to pop out, and the speed at 
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Figure 11.2 Pop-out in vision. Try to find the Y in the left panel and the O in the 
right panel. It is much easier to find the O because it “pops out” in our vision, whereas the 
Y requires a serial search through the display. Of course, once you know where to look, 
both the Y and the O are easy to find. 


1 Humans have evolved specialized areas of cortex to recognize faces. 17 This is why we see faces 
everywhere (like in the shades of light and dark on the moon as seen from the Earth). Some peo¬ 
ple have a particular difficulty with this, which is a disorder called prosopagnosia 18 (from prosopon 
[rrpoototiov, Greek for face] and agnosia [ayvtocria; a, a not + gnosis, yvcocrtc, knowing]). People with 
prosopagnosia have to rely on identifying features (a distinctive hairstyle or mode of dress) to iden¬ 
tify people. Markos Moulitsas described his prosopagnosia on his blogsite “Daily Kos,” and noted his 
inability to watch war movies (like Saving Private Ryan) or Mafia movies (like The Godfather) because 
everyone has the same haircut and is dressed the same. 19 Although it is not known what causes pros¬ 
opagnosia, Doris Tsao recently found that one of the monkeys she was studying had no face-selective 
activity in its cortex and showed an inability to recognize faces, much like people with prosopagnosia, 
so we know that this can occur naturally in other primates as well. 
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which you recognize the target does not depend on the number of distractors (because 
you have different feature detectors for the target and the distractors and you can just 
have the feature detector tuned to the target shout out when it finds it). When the target 
is really similar to the distractors (Y in a field of Xs), you have to check each one indi¬ 
vidually. This means that the time it takes you to find the target depends on the number 
of distractors. For parallel recognition (pop-out), your accuracy is independent of the 
time it takes you to respond. However, for serial recognition (searching through candi¬ 
dates), the longer you take to respond, the more accurate you will be (up to some point 
when you have gathered all the information available). This means that there is a clear 
tradeoffbetween speed and accuracy for serial recognition. 


Experts and the speed-accuracy tradeoff 

The race-to-threshold model provides a good explanation for the speed-accuracy 
tradeoff of serial search. 21 If the agent using a race-to-threshold algorithm wants to 
respond faster, it just lowers the threshold. A lower threshold means that the race ends 
faster (so the agent responds more quickly) but with the lower threshold, the agent 
has had less time to analyze the information, and is thus going to be more susceptible 
to noise. Equivalently, the agent can increase the slope. In either case—lowering the 
threshold or increasing the slope—with each small piece of information you gather, 
you approach the threshold more quickly. Increasing slope and decreasing threshold 
are mathematically equivalent. On the other hand, moving your starting point closer 
to threshold is different—that’s moving your bias—guessing before the information 
arrives. If you move your bias above the threshold, you don’t even have to wait for infor¬ 
mation, you could just make the decision without information! (I suspect this is more 
common than it should be.) 

Certainly, it is possible in some tasks that you have gathered all the information you 
can, that at some point there is no new information available, and you have reached 
what is mathematically called an asymptote. In those cases, waiting longer isn’t going 
to make your decision worse, it’s just not going to make your decision better either. 
At some point, you have to act. However, in psychology experiments with noisy and 
ambiguous data (such as the television snow experiment described above), waiting is 
always better and there is a tradeoffbetween speed and accuracy. 

So why can experts act quickly? How did Ted Williams or Tony Gwynn hit close to 
40% of the pitches thrown at them? K It takes a baseball about 400 milliseconds to get 
from the pitcher’s mound to home plate. L Given that human reaction time (even for the 
fastest athletes) is about 200 milliseconds, 23 one has only a couple of hundred millisec¬ 
onds (a couple of tenths of a second) to integrate the information about how the ball is 
spinning, where it’s going to cross the plate, and when it’s going to be there. Players with 
better reaction times have better batting averages. 


K Ted Williams and Tony Gwynn were two of the greatest hitters of all time, getting on base an aver¬ 
age of 48.4% and 38.8% of the time, with lifetime batting averages of0.344 (34.4%) and 0.338 (33.8%) 
respectively. 22 

L A 100-mile-per-hour pitch traveling 60 feet 6 inches takes about 400 milliseconds. 
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Experts have two advantages over nonexperts—they have trained up their percep¬ 
tual system to recognize better features about the task and they have trained up their 
habit system to cache the most reliable choices of what to do. Perceptual training is 
like the chunking that we saw in Chapter 10. Experts have learned how to represent 
the task in such a way that it is easier to determine the correct responses from the avail¬ 
able inputs; they have identified good categories for the task. The classic examples of 
this are chess and videogames. We have already seen how a computationally expensive 
search algorithm can beat a grandmaster at chess (Deep Blue vs. Gary Kasparov), but 
that chess grandmasters don’t actually do that same computationally expensive search. 24 
Instead, chess grandmasters have learned to represent the board in ways that limit the 
search they have to do. The experiment that proved this was that chess masters are better 
at remembering the positions of pieces on boards when those positions can arise from 
legal moves than when they can’t. Just as an expert musician no longer thinks about the 
fingering for a chord on a guitar, a chess master no longer looks at each piece on the 
board as an individual thing. Playing videogames produces a similar specific expertise— 
faster recognition of the events in the game (training perceptual recognition systems, 
this chapter) and faster recognition of what to do in response to those events (training 
procedural systems, Chapter 10). 25 

It is important to recognize that this is not saying that “intuition” is better than “analysis,” 
even for experts. Real experts shift from intuitive decisions to analytical decisions (from 
procedural to deliberative) as necessary. Similarly, it is important to recognize that rawintu- 
itionis not reliable in these situations—experts have spent years (often decades!) training 
their perceptual and procedural systems. In a sense, what the procedural decision-making 
system we examined in Chapter 10 does is trade perceptual and deliberative time during 
the decision for time during training, so that, while it takes longer to learn to do the task, 
once trained, the task can be done more quickly, if less flexibly. Typical estimates of train¬ 
ing time for most human expert-level tasks is about 10,000 hours. 26 (That’s five years at 
40 hours per week at 50 weeks per year, or four years at 50 hours per week for 50 weeks 
per year.) This is why most apprenticeships, such as graduate school, medical school, time 
in the minor leagues, etc., are about four to sixyears. M 

It is important to recognize that the opposite of “analytical” is not “intuitive.” Both 
analytical (deliberative) decision-making and intuitive (procedural) decision-making 
arise from systems that have evolved to make accurate decisions in the face of integrat¬ 
ing information. The opposite of “analytical” is “glib,” moving the threshold down so far 
that no information is gathered (or moving the bias above the threshold), both of which 
are disastrous. In fact, both analytical and intuitive responses show speed-accuracy 
tradeoffs. One key to Tony Gwynn’s success is that he used a very light bat and waited 
until the last moment to hit the ball. 27 One of the big differences between Ted Williams, 
Tony Gwynn, and the rest of us is that Williams and Gwynn had much better eyesight. 
When Tony Gwynn retired, he commented to a reporter that his eyes were going and 
he couldn’t pick up the ball as early as he used to be able to. When the reporter asked 
what his eyesight was now, Gwynn responded that it was now about 20/15, but at one 


M This of course reminds us of the old joke—our well-dressed protagonist stops a passerby on a 
New York City street and asks, “How do I get to Carnegie Hall?” The inevitable answer: “Practice, lots 
of practice!” 
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time had been 20/10. With better eyesight, he could pick up the rotation of the ball 
earlier, giving himself more time to react, more time to integrate information, and a 
better chance of hitting the ball. 

Books and papers for further reading 

• Jan Lauwereyns (2010). The Anatomy of Bias. Cambridge, MA: MIT Press. 

• Joshua I. Gold and Michael N. Shadlen (2002). Banburismus and the brain: 
Decoding the relationship between sensory stimuli, decisions, and reward. Neuron, 
36, 299-308. 

• Joshua I. Gold and Michael N. Shadlen (2007). The neural basis of decision making. 
Annual Review of Neuroscience, 30, 535-574. 
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the stone above weighs on you 
the tension in your voice 
betrays you 

human arms are weak 
but our legs are strong 
they will raise us up 


Both deliberative and procedural decision-making require recogni¬ 
tion of the "situation" we are in. Situation recognition is a catego¬ 
rization process in which we recognize that this moment shares 
similarities with previous situations we have experienced and differs 
from others. Although models exist of static situation-recognition 
processes, in actuality, situation recognition must be about recog¬ 
nizing sequences of events (narratives). The neural mechanisms 
underlying narrative remain unknown. 


Animal psychology experiments tend to be run in cue-poor environments. A rat in a 
small box with nothing but a hole to poke its nose into and maybe a few levers to push, 
or a monkey in a small chair facing a blank screen, has few choices to decide between. 
When the tone sounds and food is delivered into the food port, there’s not a lot of ambi¬ 
guity about what the new cue was. When it happens a few more times, the animal quickly 
recognizes that the tone precedes food delivery. When a dot appears on the screen, the 
monkey looks at it—it’s the only thing that has changed in the environment. 

Even so, animal behavior has to be shaped to an environment. 1 One of the most 
famous experiments used in neuroscience today is the Morris Water Maze, 2 which con¬ 
sists of a large (several meters wide) pool of water made opaque with a small amount of 
water-soluble nontoxic white paint. In this task, rats or mice are trained to find a plat¬ 
form hidden under the water. This task is popular because it is particularly sensitive to 
damage to certain brain structures, such as the hippocampus. A From a decision-making 


A As with any clinical experiment, interpretation of lesion studies needs to be tempered by an under¬ 
standing of logical implications. Damage to the hippocampus leads to problems in the Morris Water 
Maze, but it is not true that problems in the Morris Water Maze imply damage to the hippocampus. 
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perspective, the Morris Water Maze is very interesting because the task can be solved 
by either the deliberative or procedural (habit) systems, depending on the train¬ 
ing sequence. 3 But what is the first thing that rats do on this task? They swim around 
the edge of the pool, looking for a way out. Only after first thoroughly exploring this 
possibility do they start heading for the platform. 

With humans, the important cues in psychology and neuroscience experiments are 
often explicitly identified to the subject: “You will see a set of faces. Try to identify which 
faces you have seen before...” Even when they are not explicitly identified, although the 
environments for human psychology experiments are often cue-rich, the important cue 
is often obvious to the subject. An fMRI machine is a claustrophobic space with tight 
walls around you, a metal cage around your head, and extremely loud pounding thump¬ 
ing noises. 4 And, yet, when the visual cue is turned on, we know to attend to it. In part, 
we know to attend to it because we have a storyline for what we expect to happen in one 
of these experiments. 

But, as I’ve pointed out before, life outside of psychology experiments is made up of 
a plethora of cues impinging on our senses at all times. We are not often sitting, staring 
at a screen, trying to figure out if a small subset of a set of randomly moving dots tend 
to be moving more in one direction than another. Certainly, there are times that we see 
people peering into the distance or the darkness (think of a sailor standing on the crow’s 
nest on a ship, keeping a lookout for land, or of a hunter stalking a deer); however, most 
of the time, the issue is too many cues, not too few. 


Situation recognition 

I first got interested in the question of situation recognition when talking to one of my 
colleagues (Steve Jensen) over lunch. Steve was fascinated by a phenomenon of one-trial 
learning in ICD patients. 5 These patients have a risk of sudden heart arrhythmias (where 
the beating of the heart changes to the wrong rhythm) and have had an implantable 
cardioverter-defibrillator (ICD) attached to their heart. These devices detect the rhythm 
of the heart and, if that rhythm goes wrong, provide an electrical shock to restart the 
heart, (just as an external defibrillator can shock the heart back into rhythm, so can an 
internal one.) Unlike a pacemaker, which provides a slow, steady rhythm to the heart, 
the ICD kicks in only when the rhythm is stopped or wrong. Also, unlike a pacemaker, 
the ICD hurts like hell when it kicks in. Unlike an external defibrillator, which is used 
only on unconscious patients, the ICD usually kicks in while the patient is conscious. 
In such situations, the patient is suddenly provided with an extremely negative experi¬ 
ence (even if it potentially may have saved the patient’s life). These ICD patients have 
the strangest negative associations with things in their lives. Steve wanted to know what 
made the patient decide that the fork was the culprit and not the spoon, or that it was the 
act of sitting in a chair and not laughing with one’s friends. Clearly, it has something to 
do with what these subjects were attending to at the time, but it got us thinking about 
the question of how we decide what cues are important. 

In the robotics, computer science, and psychology literatures that study reinforce¬ 
ment learning, this is called the credit assignment problem. 6 Generally, the assumption 
in these models and experiments is that there is a small subset of potential cues that 
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could be important, and the agent (the animal or the computer simulation) has to figure 
out which one is the key to making the right decision. But we know from other experi¬ 
ments (such as the Morris Water Maze and other spatial navigation experiments) that 
it is often a constellation of cues that informs the animal. 7 A subset of the cues can be 
removed and the animal will still find its way correctly, as long as enough of the cues are 
available. No one cue is key. 

We realized that most of the computer models of decision-making assumed that 
the agent knew which possible situations were available (technically known as the state 
space ) and often knew how they changed from one to another (technically known as the 
transition matrix ), but that in reality, animals created situations out of cues. Real animals 
(including humans) categorized cues into situations . 8 Simple categorization in neural 
systems is well understood and can be modeled with content-addressable memories. 9 In 
fact, there are very well-studied learning systems that automatically categorize inputs, 
even without supervision. 10 These systems are called unsupervised learning algorithms 
and work by identifying regularities in the inputs. 

Using a standard unsupervised learning algorithm, we built a model that incorporated 
a situation-categorization component into a standard reinforcement learning agent (an 
agent that learned what to do in what situation, essentially a procedural-learning agent in 
the terminology used in this book). 11 We found that this situation-recognition compo¬ 
nent turned out to change the observed psychology of our agent in a number of impor¬ 
tant ways. Most importantly, disappointment (lack of delivered reward) became different 
from punishment (delivery of a negative stimulus). In our model, disappointment led to 
the recognition of a change in the situation, while punishment led to a new association. 
(We’ve talked about the differences between reward, reinforcement, and disappoint¬ 
ment, and their negative counterparts [punishment, aversion, and relief ] in Chapter 4.) 

This model suggests that there is a significant context effect to recognizing changes 
in our expectations of reward or punishment. This can explain why, for example, addicts 
who become clean in hospitals relapse on being released back into their normal lives, or 
why students who learn the correct answers to give about electricity in physics class are 
still unable to correctly build circuits when given a set of real batteries and other electri¬ 
cal components to build with. 12 

We were also able to manipulate parameters in our model to produce simulations 
of problem gambling due to incorrect situation recognition, either because the agent 
thought it could control a random situation (the illusion of control, believing that two 
outcomes were due to two situations when they really occurred due to random chance 
within a single situation) or because the agent was unable to recognize a change in a situ¬ 
ation (which led to the phenomenon of chasing losses, continued playing even in the face 
of devastating loss because the agent thinks the difference is just random chance). 13 

Although our model was able to explain a number of psychological phenomena (such 
as why behavioral extinction is not the same as forgetting or why animals are slower to 
extinguish variably delivered rewards) and a number of clinical phenomena (such as 
context-related relapse in addiction or the illusion of control in gambling), 14 the unsu¬ 
pervised learning algorithm we used was a static cue-categorization mechanism. The 
algorithm took the currently available set of cues and determined whether it was part of 
a known situation or whether it had to be declared a new situation. 13 The real world is, 
of course, much more complex. 
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Situation recognition is complex, dynamic, and hierarchical 

Of course, situation recognition is not all or none: situations are both similar and dis¬ 
similar to other situations. Making correct decisions requires that we be able to rec¬ 
ognize both the similarities and differences between situations. The usual statement 
attributed to the Greek philosopher Heraclitus is that “You never step into the same 
river twice,” but the fragments we have from the actual Greek suggest that a better 
interpretation is that some things stay constant through change. 16 If you are going to 
learn, you have to have some constancy between your past and your future. You need to 
recognize the similarities between your past and your present in order to translate past 
successes and failures into present plans. To solve this paradox, we need a system that 
can categorize situations, recognizing both the similarities and differences between 
your past and present. 

I give a lot of scientific presentations to lots of different audiences. When I build my 
slides for a presentation, I try to make sure I have the right slides for the given audience. 
I remember the first time I went to give a talk in Japan. My wife pointed out that most 
Japanese scientists could read written English much better than they could understand 
spoken English. So I added a section to each of my slides describing the basic results 
on that slide. (I have since found that this helps native English speakers as well.) But 
even though the situation was slightly different, I didn’t think of giving-a-talk-in-Japan 
to be a new situation, I defined it as a special case of my giving-a-talk category. This 
hierarchical nature is a key to human categorization. 17 A penguin is a bird, but it is also 
a water-swimmer, and it doesn’t fly. Just as we categorize our concepts in a complex 
hierarchy with special cases that do not quite fit the categories, we also must categorize 
situations in complex ways. 

Situations are also dynamic. We do not simply see a set of cues, categorize them as 
a situation, and respond to it. Instead, we recognize a sequence of situations and travel 
through it. I travel through the Minneapolis-St. Paul airport (MSP) approximately once 
per month (sometimes more, but I try to restrict myself to traveling less than once a 
month). Sometimes I have my laptop, sometimes I don’t. Sometimes I have my family 
with me, sometimes I don’t. Sometimes I have checked a bag, sometimes I take only 
carry-on. Going through security with my laptop is different from not. Similarly, secu¬ 
rity is different if it’s just me or if I have my kids, and it’s different if my toiletries are in 
my carry-on bag or checked through. And yet, I didn’t need to relearn how to go through 
security from scratch the first time that they told me I had to take my toiletries out of my 
carry-on and pass them through the X-ray machine separately. I was able to take what I 
knew about going through airport security and modify it to include taking my toiletries 
out of my carry-on suitcase. 

Going through airport security is a good example of a dynamic situation, what 
might be called a script. 18 The actions at each stage are very different, and yet they all go 
together in a recognizable sequence. First, you need your ID and boarding pass. Once 
those are checked, you then need to wait in line for the plastic tubs to put your stuff 
into. In American airports, you need to take off your shoes. (I once started to take off 
my shoes in a European airport and the security agent looked at me with quizzical sur¬ 
prise and asked, “What are you doing?”) Going through airport security is a sequence 
of subsituations and actions. 
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A script is a storyline, a categorization of a sequence that can be modified at its sub¬ 
stages. The important points here are threefold. First, we categorize the situations of 
the world in order to act within it ( What are the important cues here?). Second, those 
categories are complex and hierarchical ( Giving a talk in Japan is a special case of my 
giving-a-talk category ). Third, these scripts are dynamic ( Going through airport security is 
a complex, dynamic sequence of situations and actions ). These scripts are narratives, stories 
we tell ourselves about our lives. 


Narrative and literature 

The success of great literature is usually due to the recognizability of the characters. 
We ask ourselves, “Is that what this character would do in this situation?” Playwrights 
often talk of “keeping things in character.” When a person does something that is “out 
of character,” everyone notices. As Fergus (played by Stephen Rea) explains to Dil 
(played byjaye Davidson) in Neil Jordan’s The Crying Game, “It’s in my nature.” And 
then, to fully explain his actions to her, he tells her the fable of the frog and the scor¬ 
pion. We recognize the self-consistency of his actions as a definition of his nature. This 
is true whether the stories be realistic or fantastical. Even explicitly unrealistic plays 
are based, fundamentally, on recognizable characters. I remember the first time I saw 
Samuel Beckett’s Krapp's Last Tape and realized it was really about an old, alcoholic 
writer reminiscing over a long-past love affair. For all of its existentialist implications, 
Jean-Paul Sartre’s No Exit is a simple dramatic interaction between three very real peo¬ 
ple. Even the absurdist plays like Eugene Ionesco’s Rhinoceros have recognizable human 
interactions throughout. 

In fact, throughout literature, the stories that survive are the ones in which we rec¬ 
ognize human characteristics. Whether this be the remarkably human characteristics of 
the Greek or Norse gods, or the foibles of Biblical characters, the stories that survive are 
the ones that capture something “true” about human nature. 

In large part the success of literature arises from the fact that human nature hasn’t 
changed in thousands of years. We recognize ourselves in the squabbling Greeks 
of Homer’s Iliad and the recovering soldier working his slow way home in Homer’s 
Odyssey. We recognize ourselves in Hamlet’s indecision, in the rashness of King Lear. 
The soap opera of Shakespeare’s A Midsummer Night’s Dream is recognizable in any 
high school clique. The fables of Africa, North America, and medieval Italy all speak to 
us reliably . 19 We can even find ourselves in the imagined worlds of fantasy and science 
fiction, whether it be the hobbits learning to stand tall in J. R. R. Tolkien’s Lord of the 
Rings or the political machinations in Frank Herbert’s Dune. 

It is important to recognize that this doesn’t mean that literature is a morality play; 
true morality plays tend to be boring and quickly forgotten. Successful literature is more 
like an experiment, a way of testing “what would happen if.” Literature is a form of Ac¬ 
tive learning, a way to see what would happen if someone with a certain character were 
faced with a specific situation. It allows us to ask ourselves, “Would I do that?” Whether 
we ask ourselves if we would make the same mistakes or if we would have the strength 
to take the right action, literature is a way of trying out hypotheses about consequences, 
much like the Gedanken experiments (thought experiments) in science. 
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Writing instructors tell nascent writers to find the character, and then to put him or 
her into the most difficult situation one can. 20 What makes a situation difficult depends 
on the character one is writing about. At each stage of a story, we ask ourselves, “What 
would this character do in this situation?” We are very good at recognizing forced fiction, 
where the characters do something that is out of character to further the plot. 

Fundamentally, humans are narrative creatures. 21 We tell ourselves stories of our lives 
and we define our lives in terms of those stories. We use narrative to explain our past, 
to recognize our present, and to predict the future. We plan the future by telling stories 
of what we think will happen (see Chapter 9). We recognize our present through the 
scripts we are passing through, and we try to understand our past actions and the things 
that have happened to us in terms of a narrative structure, by telling stories. Of course, as 
we saw in Chapter 9, that past is constructed—reconstructed, one might say. By chang¬ 
ing the stories we tell ourselves, we reinterpret why we did what we did and we change 
who we are. 

For example, the simple sentence “There is no such thing as one drink for an alco¬ 
holic” implies that the decision options you have in the future are not between going to 
the bar and having a single drink or staying home and being bored, but rather between 
going to the bar and getting really drunk (and thus regretting it) or staying home and 
not. This is a process called “bundling” in the neuroeconomics literature. 22 By bundling 
future decisions together, one sees both the immediate and long-term consequences. 
Computational models of this show that these bundling effects can change decisions in 
ways that nonbundled decisions cannot. 

At this point, we don’t know very much about the neurophysiology of how stories 
are represented within the brain. Clearly it must involve representations of categories 
and content-addressable memories. We know from the psychological literature that 
they must exist, and we know something of how static categories are represented within 
the brain, but the complexity of situation-recognition systems (such as the hierarchical 
nature) and the dynamics of those representations (such as the narrative structure and 
the effect of that narrative structure on decision-making) are still very open questions 
in the scientific literature today. 

Books and papers for further reading 

Rather than point you to scientific papers about the role of narrative, I’m just going to 
suggest reading fiction. Great authors are keen observers of the human condition. 8 


B Great fiction can be tremendously informative. But, as with any scientific paper, one has to engage 
the work in a dialogue and face the observations with a critical mind. 
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Tropical mountains 

through the netting of a blue hammock, 
snuggled together, 
reading, 
waking up 


The specific goals that the decision-making systems are trying to 
satisfy arise from intrinsic goal functions evolutionarily prewired 
into our brains. However, we can also learn to identify new goals 
that we have learned lead to those intrinsic rewards or avoid those 
intrinsic punishments. The vigor with which we attempt to attain 
those goals depends on the level of intrinsic need and the opportu¬ 
nity cost of chasing those goals at the expense of others. 


So far, we’ve talked about decisions as arising from evolutionarily learned reflexes, and 
from Pavlovian, Deliberative, and Procedural systems that learn from rewards and punish¬ 
ments. We’ve talked about how animals integrate information about the world and categorize 
situations to make their decisions. Throughout our discussion, however, we have sidestepped 
a fundamental question: What defines the goal? We identified opioid signaling as measuring 
the positive and negative reward and punishment components, and dopamine signaling 
as measuring the differences between predicted and expected (see Chapters 3 and 4). But 
neither of these defines what those positive rewards or negative punishments are. 

The first explicit models of decision-making assumed that the goal was to maximize 
value or “utility.” 1 Adam Smith (1723-1790) argued that everyone is trying to maximize 
his or her own utility. 2jA From the very beginning of the economic literature, it was imme¬ 
diately clear that it was impossible to define utility in a general way that would be the 


A Although Adam Smith is usually cited for the concept of each individual maximizing his or her 
own utility, it is important to note that he recognized that this was not stable in its simplest form. 
Smith’s conceptualization was actually quite complex, and he suggested that national economic success 
depended on what we now recognize as empathy, sympathy, and altruistic punishment. For example, 
he recognized the importance of governmental assurances of market fairness, as well as the dangers of 
monopolies, and other complexities beyond the simplicity of maximization of individual profit . 3 
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same for all people, and so the mathematicians Nicolaus Bernoulli (1695-1726) and his 
younger brother Daniel Bernoulli (1700-1782) proposed the idea of subjective utility, 
which suggests that each of us has an internal measure of that utility. 6 But it still doesn’t 
define how our interaction with the world is translated into rewards and punishments. 

Similarly, the original computer models of decision-making defined the goal to maximize 
as a reward function of time or situation, but never defined what the reward was. 7 These 
computer models did this to explicitly avoid the question that we are trying to address here. 
The authors of those computer models were trying to understand the decision-making 
algorithms in an imaginary (disembodied) computer world. When these algorithms were 
translated into robotics, the authors had to hard-code the goals into the robots. 8 


B The history of why the Bernoulli brothers came up with subjective utility is interesting, particu¬ 
larly because it gives an insight into the problems that arise from trying to derive microeconomics from 
first principles. 4 The brothers were trying to understand how to incorporate probability into utility— 
How much should you like a 10% chance of getting $50? They assumed, for simplicity, that one should like 
it 10% as much. Thus, 10% of $50 should be equivalent to $5. The problem with this is that you can 
create paradoxes from this assumption that are clearly wrong. 

The case that the Bernoulli brothers came up with is called “the St. Petersburg paradox” because it 
was first presented to the Imperial Academy at the court of Peter the Great in St. Petersburg. Imagine a 
game with the following rules: You pay a fee to get in and then a coin is flipped. The game stops when 
the coin comes up tails. The game pays out $1 for each coin flip. Thus, if tails appears on the first flip, 
you get $1, but if you get 99 heads followed by a tails, you get $100. The question is How much would 
you pay to play this game? Because of the way the game is constructed, the mathematically expected pay¬ 
out is infinite, and a logical (which economists call “rational”) person should be willing to pay anything 
to play this game. 

Of course, people will only pay a small amount to play this. (Compare the St. Petersburg para¬ 
dox, which feels wrong even though it is logical, to the distortions that were seen by Kahneman and 
Tversky, 5 which feel right even though we know they’re illogical, such as the Asian flu example, where 
describing the problem in terms of people saved or people lost changes the choice [see Chapter 3].) 

The Bernoulli brothers proposed that the reason people wouldn’t pay large sums to play this game 
was because they were actually maximizing “subjective utility” in which higher and higher payouts 
had less and less value. Thus, $200 was not twice as valuable as $100. Bernoulli specifically suggested a 
square root function—that is, subjective value is the square root of the actual value offered. 

However, it is not difficult to modify the St. Petersburg paradox to wreck any unbounded value func¬ 
tion. One possible explanation is that our subjective value functions are not unbounded. Eventually, 
it doesn’t matter how much money is paid out, it’s just a “big win.” The problem with this hypothesis 
is that it is very clear (at least for money) that this is not true. Billionaires who have long had every 
possible desire sated continue to fight for more money. (We will see why later in this chapter.) Current 
explanations for the St. Petersburg paradox are based on differences in how much we like gains and 
losses (people hate losses more than they like gains), and the recognition that you can never be offered 
an infinite amount of money because eventually you’ll break the bank. 

My hunch is that part of the answer may be that valuation is done online, and that the valuation 
system in our brains can’t do the infinite sum needed to make it worth paying into the St. Petersburg 
paradox. If the person adds up only the first few cases and then says the chance of 10 heads in a row is 
so unlikely it’s not worth thinking about, then the maximum payout is not infinite. (Of course, people 
play the lottery, which has a much lower probability of payout.) Even though the Bernoulli brothers’ 
explanation of the St. Petersburg paradox as a transformation of utility is no longer the accepted story, 
the concept of an internal modification of subjective utility remains. 6 
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Intrinsic reward functions 

Theorists studying how animals forage for food in the wild have argued that animals 
(including humans) are maximizing their energy gains and minimizing their energy 
costs. 9 For example, imagine a hummingbird gathering nectar from a flower. To get the 
nectar, the hummingbird hovers in front of the flower, pulling nectar out. As the bird 
drinks the nectar it becomes harder and harder to pull nectar out. At some point, the 
energy being spent to flap its wings fast enough to hover is more than the energy it is 
pulling out of the flower, and it should go to another flower. Going to another flower is 
risky because the bird doesn’t know how much nectar is available at that other flower. 
Maybe it has already been tapped out. Maybe the bird will search for a long time. It is 
possible to calculate the optimal time that the bird should spend at a patch of food and 
when it should leave. This optimal time depends on the size of the patch, the expecta¬ 
tion of how much food is left in the patch, what the expected travel time is to the next 
patch, and the likely amount of food the bird would expect to find at the next patch. 
These equations correctly predict stay and leave times for foraging and hunting birds, 
mammals, and insects, transitions in monkey decision-making experiments, even for 
human hunters and gatherers deciding when and where to look for food, and for people 
foraging for information on the Web and in the world. 10 

But these foraging theories are based on the assumption that the only purpose of 
these animals is survival. Darwins natural selection theory tells us that what really drives 
success is not survival but the procreation of genes into future generations, in particular 
successful procreation of future successful procreators. 11 There are numerous examples 
where animals sacrifice themselves for their genetic future. Even something as simple 
as a parent feeding its children is proof that there is more to the picture than individual 
survival.*" We all know examples of parents sacrificing for their children, whether it be 
penguins starving themselves to keep eggs warm, starlings bringing food back to a nest, 
a polar bear protecting her cubs, or human immigrants sacrificing their dreams so their 
children can succeed in a new land. 

Of course, there are also the classic studies of the social insects (bees, ants), who 
live in large colonies where drones sacrifice for a queen and her children. 12 The reason 
for this is now well understood, and is due to the fact that because of the way the insect 
genetics works, a drone is more genetically similar to her sisters than to her own chil¬ 
dren, so she evolutionarily prefers to make more sisters than to make more children. 

These descriptions assume that animals are actually calculating how to maximize 
their genetic success. I don’t know about you, but I don’t generally spend each morn¬ 
ing thinking, “How can I increase my genetic success today?” I’m pretty sure that no 
other animal does either. In large part, this is because determining what is going to lead 
to genetic success is very hard to predict. Such a calculation would take a tremendous 
amount of brain power—an individual spending too much time worrying about it is 
unlikely to actually succeed at it, and would quickly become genetically outcompeted. 

So what does define the goals we want? One suggestion is that we have intrinsic goal 
functions evolutionarily prewired into our brains, 13 but that doesn’t say what those goals 


I bet you never thought cooking for your kids was so fraught with meaning. 
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are. We are back to the hard-coded goals that the roboticists put into their robots. These 
authors argue that our evolutionary history has hard-coded goals into our brains that 
we then seek out irrespective of their actual evolutionary use. For example, dogs like to 
chase things, presumably because in the evolutionary past, the primary things that they 
got to chase were big game like deer, moose, and elk. For many dogs, the primary things 
they get to chase now are cars. These concepts were pushed forward in the 1970s under 
the guise of sociobiology —usually stated too simply and too unwilling to address the 
complexity of primate interactions. 14 ^ 

The origins of sociobiology were in animal ethology studies, particularly insect stud¬ 
ies. 18 While insects are often interpretable as little robots, 19 with simple routines that are 
often recognizable (moths fly in a spiral toward lights), and often with simple failure modes 
(as anyone watching a moth fly toward a flame can see), birds and mammals are more 
complex, and humans particularly so. Human failure modes are also particularly complex, 
but that doesn’t mean they’re not identifiable. (We will address many of the ways that the 
decision-making systems break down [their failure modes ] in the next section of the book.) 

Human social interactions are incredibly complex. This does not, however, imply 
that they are random, unobservable, or unidentifiable. As discussed in Chapter 12, nov¬ 
elists and playwrights have been observing humans for millennia. As an audience, we 
recognize that individual humans have a consistent character. In a novel or a play or 
a movie, if someone does something that is “out of character,” it shows up like a sore 
thumb. We have a very large portion of our brains devoted to recognizing social interac¬ 
tions. Does this mean we know which way an individual will jump? Not perfectly, but 
we can predict what an individual is likely to do, and we make our plans based on those 
predictions. 

So what are those intrinsic goal functions? Obviously, when we are hungry, we want 
food; when we are thirsty, we want water. And obviously, we desire sex. Notice that a 
desire for sex is different than a desire for children. The evolutionary development of the 
desire for sex as the means of driving procreation rather than the desire for children as 
the means for procreation is a classic example of an intrinsic reward function. Of course, 
some couples do have sex because they are trying to have a baby. But, often, sex is for its 
own pleasures, and couples are surprised at the unexpected pregnancy. 


D Sociobiology has gotten a bad rap over the past several decades because the conclusions were 
taken as prescriptive rather than descriptive, and because they often created broad generalizations from 
limited observations. 15 We will discuss the issue of differences between prescription and description 
later (in Chapter 23). Briefly, the goal of science is to understand causality in the world—if the world 
is in this configuration, then that will happen. For example, the Galilean/Newtonian theory of gravity 
implies that if I drop a hammer and a feather at the same time in a vacuum, they will hit the ground at 
the same time. (This was of course tested by Apollo 15 astronauts and found to be completely true. 16 ) 
We call this descriptive because it describes the world. On the other hand, what we should do with this 
knowledge is prescriptive. To say that the double standard of loose sexuality (women lose rank for sex¬ 
ual promiscuity, while men gain it) arises from differences in how much effort women and men have to 
put into childrearing (it is easier for men to leave than for women to) 17 does not imply that one should 
encourage that double standard, or that a society that accepts that double standard will be better than 
one that doesn’t. We will come back to questions of “better” and society in our discussion of morality in 
Chapter 23. As this issue of sociobiology is not the subject of this book, I’m going to leave it here, with 
the simple statement that the fact that biology is not destiny does not mean that it isn’t true. 
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What’s interesting^ of course, is that humans (like all animals) are actually very 
picky about sex—partners who are more likely to produce strong, healthy, and intel¬ 
ligent children are more attractive sexually. 20 In addition, because humans require long 
commitments to raise successful children, humans, particularly women, often demand 
some sort of commitment before sex. 21 There have been a tremendous number of stud¬ 
ies examining the complex interaction that arises in the human mating game, showing, 
for example, that beauty is not in the eye of the beholder, but that there are standards of 
beauty that are consistent across cultures and individuals that signal information about 
health, fecundity, and intelligence. 22 

Certainly some components of beauty do seem to vary from culture to culture, but these 
components tend to entail elaborate preparations that show that we have friends and the 
time to maintain our appearances. Because humans live in communities (it really does take 
a village to raise a child in normal human societies 23 ), many of the variable components of 
definitions of beauty require a complex interaction with others of the community to main¬ 
tain. (Think how difficult it is to cut one’s own hair.) The specifics of these preparations may 
change culturally, but they consistently indicate time and effort put into appearance. 

But food, water, and sex are not enough for us: we also want to be respected, and we 
seek out music, excitement, family, and friends. Babies seek out their mothers; mothers 
and fathers seek out their children. As a specific case, let’s take maternal interactions. In 
one of the classic histories of the sequence of science, early studies of interaction sug¬ 
gested that animals did things only for food reward. 24 This suggested to researchers that 
children were “attached” to their mothers because they were fed by them. E In particular, 
the behaviorist John Watson (who we have met before as one of the major players in 
the development of the concept of stimulus-response “habits”) argued that touching 
infants could only transmit disease and thus kids shouldn’t be touched any more than 
necessary. 25 A third of a century later, a young psychologist named Harry Harlow, setting 
up a new laboratory at the University of Wisconsin, came across a description of how 
children being separated from their mothers as they entered a hospital would scream 
and cry. A few weeks later, his rat laboratory still not ready for experiments, he watched 
monkeys playing at the Madison zoo and realized he could directly test whether this 
anxiety was due to the separation from their mothers or whether it was some gener¬ 
alized stress, like going to the hospital. Harlow began a very famous series of experi¬ 
ments about motherhood and found that infants needed the warmth of a mother (or a 
mother-analog) as much as they needed food. 26 

Although these experiments galvanized the animal rights movement 27 and are 
still held up as terrible, nightmarish experiments (and would be very difficult to get 
approved under today’s animal-care regulations 11 ), they were a key to understanding 


E This seems silly to us now, but it was due to an inability to quantify the non-food-related attach¬ 
ment component. Remember that psychology moved from the Gestalt theorists (in the late 19th 
century), who used introspection, to the Behaviorists (early 20th century), who would only examine 
quantifiable behaviors. 

r The issue of animal rights is a very complex one that is far outside the scope of this book. However, 
for those who are interested in this issue, I recommend starting from two books by Deborah Blum, The 
Monkey Wars, which describes the interactions between the animal rights movements and scientists, and 
Love at Goon Park, which describes the life of Harry Harlow and the influence of his work on the world. 
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maternal interactions. As a consequence of these experiments, hospital procedures 
were changed. For example, infants in the neonatal intensive care unit (NICU) are 
now touched regularly by a parent or nurse, which has a dramatic effect on infant sur¬ 
vival. These procedures were changed due to John Bowlby and his work on attachment 
theory. 28 Although Bowlby’s initial hypotheses were arrived at simultaneously with 
Harlow’s first experiments (1958/1959), throughout the subsequent decades, Harlow’s 
careful experimental work formed the basis for Bowlby’s successful campaign to change 
hospital procedures, as evidenced by the prominent place Harlow’s experiments had 
in Bowlby’s influential book on attachment theory. Every child that has survived the 
NICU owes his or her life in part to Harry Harlow and those terrible experiments. 0 

We now know that maternal interactions between children and their parents 
(in humans both mothers and fathers) are driven in part by the neurotransmitter oxyto¬ 
cin, a chemical released in the hypothalamus in both mothers and infants shortly after 
birth. Oxytocin seems to be a key component of the social glue that holds couples and 
families together. 32 Externally delivered oxytocin makes people more likely to trust a 
partner in economic decision games. In mothers and infants, oxytocin is released in 
response to holding children. (This was one of the points of Harlow’s work—part of 
the maternal bond is due to the social contact. 33 In fact, oxytocin both induces mutual 
grooming and is released during mutual grooming in a number of mammalian species, 
including humans.) Merely holding and playing with a child releases oxytocin, even in 
men. I still remember vividly the first time I held each of my children and fed them and 
the absolutely overwhelming emotion associated with it. Even today, I really like cook¬ 
ing for my kids and seeing them like the food I’ve made. 

Like eating, drinking, and sex, parental care is an intrinsic goal function. We don’t 
eat or drink because it’s necessary for survival, we eat and drink because we enjoy it, 
because it satisfies a need we feel in our bodies. Similarly, we don’t have sex because we 
are trying to procreate (usually). Proof that parental care is an intrinsic goal function 
can be seen in the way people treat their pets (and, perhaps, even in the fact that people 
keep pets in the first place). At one time, it might have been true that dogs were hunt¬ 
ing companions, and that cats kept granaries clean of rodents, but most households in 


G Some readers may complain about the sequence that I’ve presented here. In 1900, John Watson 
is a leading psychologist and argues that children should not be hugged or cuddled, which leads, actu¬ 
ally, to high death rates in orphanages and hospitals. 29 It takes an interaction between John Bowlby 
observing what we now identify as separation anxiety in children in hospitals and Harry Harlow doing 
terrible experiments on monkeys to overturn this view and recommend that children need (as the 
now-popular saying goes) “eight hugs a day.” It took a half-century for science to correct this mistake. 
What errors are we facing today? What scientific mistakes are hidden in this book? Should we trust 
current science at all? This topic is too long and complex to address in this book, but the short answer 
is that science is a self-correcting system. One could argue that self-correction is the key to science—it 
may take years or even decades, but science checks itself and corrects itself. 30 That’s what has allowed 
all of the breakthroughs we’ve seen today, from flight to computers. Newton corrected Aristotle, and 
Einstein corrected Newton. Instead of saying “science is sometimes wrong, we should ignore it,” we 
need to identify where we think current theories are mistaken and propose experiments that will 
directly test them—and then trust the answers those experiments give us. (Compare this to alternative 
description-of-the-world paradigms that claim to have known the answer for hundreds or thousands of 
years, but have no means of changing hypotheses in response to their relation to facts. 31 ) 
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urban and suburban settings aren’t hunting for survival, nor do they have granaries to 
be kept rodent free. H Nevertheless, more than half of U.S. households have pets of one 
sort or another. People with pets live longer and are more happy than people without. 34 
Pets, particularly dogs and cats, capture some of that intrinsic parental care goal func¬ 
tion. Just as dogs chase cars because they love to chase things, we have pets because we 
need that social interaction. Dogs have been shown to be therapeutic stress relievers for 
both traumatized children and for college students studying for finals. In college, during 
finals week, local groups would come around with their therapy dogs. I can say from 
personal experience that there is something remarkably relaxing about taking a break 
from studying to play with a big St. Bernard. 

So what are the intrinsic goal functions of humans? The list would probably be very 
long and may be difficult to pin down directly, in part because some of these intrinsic 
goal functions develop at different times. It is well known that interest in the opposite 
sex fluctuates from not even noticing differences in the very young (say younger than 
seven years old), to not wanting to talk to the other sex (the famous “cootie” stage, say 
seven to eleven), to intrigue (twelve, thirteen), and eventually to full-blown obsession 
(say seventeen). Maternal and paternal interests also famously switch on at certain ages. 
But all of these intrinsic goal functions contain some learning as well. Although teenage 
girls are often obsessed with babies, teenage boys are often disdainful. However, teen¬ 
agers, particularly teenage boys, with much younger siblings are often more parental 
(in the sense of having positive reactions to very young kids) than teenagers without 
younger siblings. 

Many of the intrinsic goal functions in humans are cognitive. One of the most inter¬ 
esting that I find is the satisfaction in completion of a j ob well done. There is an inherent 
feeling of success in not just the completion, but the knowledge that the job specifically 
was well done. 35 The act of discovery is itself a remarkably strong driving force. Although 
there have been those who have argued that curiosity is really due to issues of informa¬ 
tion gathering needed for proper behavior within an environment, several recent com¬ 
putational studies have found that an inherent drive to explore produces better learning 
than simple reward-maximization algorithms and that humans and other animals seem 
to have an over-optimism producing exploration. 36 Even other primates will work 
harder for curiosity than for food rewards. 37 Many scientists (including myself!) have a 
strong emotional feeling of accomplishment when they identify a new discovery—we 
now know something that was not known before, and that emotion of accomplishment 
is a strongly motivating force. 38 

Similarly, many of the clear intrinsic goal functions of humans are social, the respect 
of one’s peers (as, for example, sports stars who want to be “the highest paid wide 
receiver” or celebrities who can’t give up the limelight). This respect is often incor¬ 
rectly called social “rank,” but primate societies are very complex and differ from spe¬ 
cies to species. 39 For example, baboon societies include parallel hierarchies of females 
and males, 40 and a remarkable reverse hierarchy in the males, where the most success¬ 
ful males (the ones who make decisions that everyone follows and the ones who have 


H Although, when my wife and I were in college, we had an apartment kept mouse-free by our 
excellent hunting cat Koshka. When we moved in, there were mice living in the furnace room, but they 
didn’t last very long. 
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the most matings and the most children) are the ones who lose the fights (because they 
walk away). 41;I 

Human societies are particularly complex and leadership roles vary with situation and 
context. 42 In small bands and tribes, social rank can be very dynamic. Individual expertise 
provides transient leadership roles and corresponding rank increases. In one of my favorite 
examples of this, for her Master’s thesis, my wife watched children in the Carnegie Museum 
in Pittsburgh. 43 Some children are dinosaur experts, while other children are not. In the 
dinosaur section of the museum, dinosaur-expert children were treated with more respect 
and deference than their peers. As a control, she also watched those same children in the 
geology section, where their expertise no longer gave them that rank boost. Of course, 
rank can bleed over from ones expertise to other aspects of life. The dinosaur-expert chil¬ 
dren were more likely to be asked to make unrelated decisions (such as where to go for 
lunch) than the nonexpert children. We see this, for example, in celebrities (such as actors 
or TV pundits) asked to comment on decisions as if they were scientific experts. 

Neurophysiologically, many studies have found that these intrinsic goal func¬ 
tions increase activity in the dopamine systems (remember, dopamine signals val¬ 
ue-prediction error and serves as a training signal) and increase activity in a specific set 
of ventral structures known to be involved in pleasure, displeasure, and drive. Generally, 
these studies have asked what turns on these ventral decision-making structures, includ¬ 
ing both the ventral striatum, which sits underneath the rest of the striatum, and the 
insula (which also sits at the ventral side of the brain, but more anterior, forward, than 
the ventral striatum) . 4+ Although these studies can’t answer whether these functions are 
intrinsic or learned because they have to be studied in adults, the studies find an intrigu¬ 
ing list of signals that drive this activity, including food, money, drugs, attractive faces, 
successfully playing videogames, music, social emotions such as love and grief, even 
altruism (e.g., giving to charity) and justice (e.g., punishing unfairness). 45 

In particular, humans seem to have hard-coded goals that encourage positive social 
interactions, particularly within defined groups. 46 Humans seem to have an intrinsic 
goal of a sense of fairness, or at least of punishing unfair players. In one of the more 
interesting studies, people interacted with two opponents, one of whom played fairly 
and one of whom didn’t. 47 The people and the two opponents were then given small 
(but unpleasant) electric shocks. The key question in the experiment was the activa¬ 
tion of the ventral reward-related structures of the subject watching the two opponents 
get electronic shocks. In both men and women, there was more activation in the insula 
(signaling a negative affect) when the fair opponent got shocked than when the unfair 
opponent got shocked. In men, there was even activity in the ventral striatum (signal¬ 
ing a positive affect) when the unfair opponent got shocked. Part of the intrinsic goal 
function in humans is a sense of fairness in our interactions with others. We will see this 
appear again in our discussion of morality and human behavior (Chapter 23). 


1 For those who are interested in this, there are a lot of great books on the topic of primate hierar¬ 
chies and their complexity. See, for example, A Primate's Memoir (Robert Sapolsky), Sex and Friendship 
in Baboons (Barbara Smuts), Almost Human (Shirley Strum), Chimpanzee Politics (Frans de Waal), 
and of course, the entire series of Jane Goodall’s popularizations of her studies of the chimpanzees of 
Gombe (In the Shadow of Man, Through a Window, and many others), 
t It is remarkable how unpleasant even a small electric shock can he. 48 
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In the earlier chapters on decision-making, we discussed how we make decisions 
based on our goals, but didn’t identify what those goals were. We have mentioned some 
of the goals here, but the goals vary in large part from species to species and from indi¬ 
vidual to individual within a single species. Identifying the goals that generalize across 
human behavior and the distributions of goals that vary between humans are both areas 
of active research today. 49 Some of those goals are learned over evolutionary timescales 
and are hardwired into an individual, while others are sociological in nature, depend on 
inherent processes in our cultural interactions, and must be learned within the lifespan, 
and some are due to an interaction between genetics and culture. 50 The key point here, 
however, is that although goals are evolved over generations and learned over time in 
such a way that they have helped procreate our genetics, the goals that are learned are 
only indirectly aimed at successful procreation. Whether it be dogs chasing cars or col¬ 
lege boys surfing the Internet for porn, it is the intrinsic goals that we are trying to sat¬ 
isfy. The decision-making systems that we have explored have evolved to satisfy these 
goals. In later sections, we will see how these indirect intrinsic goals can lead us astray. 


Learning what’s important 

Hunger informs us when we need sustenance, thirst when we are dehydrated, but people 
also eat when they are not hungry and drink when they are not thirsty. Actually, that’s 
not the full story—a more careful statement would be that we can become thirsty even 
when we are not dehydrated, and hungry even when we don’t need food. This occurs 
because certain cues (visual cues, odor cues) can drive us to become hungry and to 
desire food or to become thirsty and to desire water. 

The smell of baking bread, an image of pizza flashed on the screen, cues that remind 
us of food can make us hungry. Animals that encounter a cue predictive of food are often 
driven to eat. 51 Similarly, cues that appear before something bad make us afraid. 52 We’ve 
already seen how these cues can lead us to Pavlovian action-selection, which we saw in 
Chapter 8 primarily involved emotions and somatic actions. In addition, these cues can 
create motivation that drives the other action-selection systems. 53;K 

These effects are known as Pavlovian-to-instrumental transfer (PIT) because they 
involve training similar to the learning we saw in Pavlovian action-selection systems 
(Chapter 8) and an influence on the other two action-selection systems (Deliberative 
and Procedural, which were historically known as “instrumental” learning; see 
Footnote 6C). 55 In PIT, as in Pavlovian action-selection, two cues are presented such 
that one is informative about whether the other is coming (Pavlov ringing a bell means 
food will be delivered soon). However, here, instead of driving action-selection (sali¬ 
vate), the learned contingency changes your motivation, your desire for that outcome. 
In a sense, the bell makes the dog think of food, which makes it hungry. 

Scientists studying the effects of marketing have suggested that restaurant icons 
(such as the “golden arches” of McDonald’s or the instantly recognizable Coke or Pepsi 
symbols) have a similar effect on people, making them hungry (or thirsty) even when 


K Several authors have suggested that this is what craving is—the motivation for a thing predicted by 
cues. 54 We will return to the question of craving in our discussion of addiction (Chapter 18). 
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they are not. Presumably this only works on people with positive associations with 
these symbols. It might be very interesting to measure the emotional associations with 
certain fast-food restaurants (which tend to produce strongly positive emotions in some 
people and strongly negative emotions in others) and then to measure the effects of their 
symbols on choices, but, to my knowledge, this has not yet been done. However, Sam 
McClure, Read Montague, and their colleagues found that individual preferences for 
Coke or Pepsi were reflected in preferences for identical cups of soda labeled with differ¬ 
ent labels. 56 They found that the label changed what people liked—Pepsi-liking people 
preferred the Pepsi-labeled cup, even though the two cups had been poured from the 
same bottle. Similarly, Hilke Plassmann and her colleagues studied effects of pricing on 
the perception of wine and found that people thought higher-priced wines tasted better, 
even when they were given identical wines in expensive and inexpensive bottles. S7 

The influence of this Pavlovian learning of contingencies (cue implies outcome) on 
instrumental action-selection (Deliberative, Procedural) can be either general, leading to 
increased arousal and an increase in speed and likelihood of taking actions overall, or spe¬ 
cific, leading to increased likelihood of taking a specific action. In generalized PIT, a cue 
with positive associations produces a general increase in all actions, while in specific PIT, 
the cue produces an increase in actions leading to the reminded outcome but not to other 
outcomes. 58 We will address generalized PIT in the next section, when we talk about 
vigor and opportunity costs; in this section, I want to look at the effects of specific PIT. 

Imagine, for example, training an animal to push one lever for grapes and another 
lever for raspberries. (Simply, we make the animal hungry, provide it levers, and reward it 
appropriately for pushing each lever. Of course, in real experiments, these are done with 
flavored food pellets, which are matched for nutrition, size, etc. so that the only difference 
is the flavor.) Then, imagine training the animal (in another environment) that when it 
hears a tone, it gets a grape. (Simply, we play the tone and then quickly thereafter give the 
animal the grape. Do this enough, and the animal will expect that getting a grape will fol¬ 
low the tone.) Putting the animal back in the lever environment, when we play the tone, 
the animal will be more likely to push the lever for grapes. We can test for generalized 
versus specific PIT by examining whether the animal presses both levers (a generalized 
increase in activity) or only the grape lever (a specific increase in the action leading to 
getting grapes). What’s happening here is that the animal is reminded of the possibility of 
getting grapes by the tone, which leads to an increase in the likelihood of taking an action 
leading to getting grapes. In a sense, the tone made the animal hungry for grapes. 

Of course, this also happens in humans. Seeing the paraphernalia for a drug that an 
addict tends to take leads to craving for the drug. 59 Some cigarette smokers prefer to self- 
administer a nicotine-free cigarette rather than an intravenous injection of nicotine, 60 
suggesting that the conditioned motivational cue can be even stronger than the addic¬ 
tive substance. And, in the examples we keep coming back to in this section, marketing 
symbols increase our internal perceptions of hunger and thirst. 61 I took my kids to a 
movie theater last week, and, while sitting waiting for the movie, my son complained 
that all of the food ads L were making him hungry, even though he had just eaten a large 
lunch. He said, “I wish they wouldn’t show those ads, so I wouldn’t get hungry because 
I’m already so full.” 


Some of the ads were for pizza! Who eats pizza at the movies? 
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Vigor—Need and opportunity costs 

Most of the choices in our lives entail doing one thing at the expense of another: if we 
are working at our desk, we are not exercising; if we are sleeping, we are not working.... 
There are dozens if not hundreds of self-help books aimed at time management. The 
losses that come from not doing something can be defined as an opportunity cost 61 —what 
opportunities are you losing by choosing Option A over Option B? 

Yael Niv and her colleagues Nathaniel Daw and Peter Dayan recently suggested that 
the average (tonic) level of dopamine is a good measure of the total reward available in 
the environment. 63;M This suggests that tonic dopamine is related to the opportunity 
cost in the environment. An animal being smart about its energy use should change 
the vigor with which it works as a function of this opportunity cost, (if there are lots of 
other rewards in the environment, one should work faster, so one can get to those other 
rewards quicker. On the other hand, if there are not a lot of other rewards, then don’t 
waste the energy doing this job quickly.) 

Interestingly, this tonic-dopamine-as-vigor hypothesis unifies the role of dopamine 
in learning (phasic bursts drive learning), incentive salience (dopamine implies a 
cue that leads to reward), and the motoric effects of dopamine in Parkinsons disease, 
Huntington’s disease, and other dysfunctions of the basal ganglia. 66 

Parkinson’s disease occurs when the dopamine cells die, leaving the person with a 
severe deficiency in dopamine available to the basal ganglia. 67 (Remember from Chapter 
10 that the basal ganglia use dopamine to learn and drive Procedural learning.) For 
years, dopamine was thought to drive movement because excesses in dopamine to the 
basal ganglia produced inappropriate movements (called dyskinesias, from the Greek 
Suer- [ dys -, bad] + Kivqcrla [ kinesia , movement]), while deficiencies in dopamine to the 
basal ganglia produced a difficulty initiating movements and a general slowing of reac¬ 
tion time (called akinesia or bradykinesia, from the Greek a [a-, without] and the Greek 
PpaSu- \brady-, slow]). 68 

The other thing that drives motivation and the willingness to work is, of course, need. 
As we discussed earlier in this chapter ( Intrinsic goal functions, above), a major role of 
motivation is to identify your needs—a state of dehydration means you need water. This 
is almost too obvious to say, but it is important to remember that these are not binary 
choices—it is not “Am I hungry?” but “How hungry am I?” The hungrier you are, the 
more willing you are to try new foods, the more willing you are to work for food, and the 
better that food tastes. Vigor also depends on how badly you need that outcome. 69 

Several authors have suggested that dopamine levels drive invigoration, from both 
the opportunity cost and need perspectives. 70 Basically, these authors suggest that tonic 
dopamine levels enable an animal to overcome effort—the more dopamine available, 
the more the animal is willing to work for a result. Intriguingly, this has evolutionary 
similarities to the role of dopamine in very simple vertebrates, where dopamine invigo¬ 
rates movement. 71 


M Remember that phasic bursts of dopamine signal transitions in value being better than expected, 
while pauses of dopamine signal transitions in value being worse than expected. 64 These phasic hursts 
are like taking a derivative of the expected value signal. Tonic dopamine would then be taking the inte¬ 
gral of this value signal, which is the average reward in the environment. 65 
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Manipulations of dopamine can drive an animal to work more for a specific reward 
cued by the presence of visual or auditory signals when that dopamine is increased. 72 
The specific interactions between dopamine, learning, invigoration, and reward is an 
area of active research in many laboratories around the world, and a source of much 
controversy in the scientific literature. 73 Most likely, dopamine is playing multiple roles, 
depending on its source, timing, and neural targets. 

In any case, how willing an animal is to work for reward or to avoid a punishment is an 
important motivational issue. 74 Vigor is very closely related to the issue of general PIT dis¬ 
cussed earlier in this chapter. The presence of cues that signal the presence of rewards likely 
implies that the world around you is richer than you thought. If there are more rewards 
around to get, working slowly means spending more of that opportunity cost, and a smart 
agent should get the job done quicker so that it can get back to those other rewards. 


Summary 

In previous chapters we talked about action-selection, but not about what motivated 
those actions. Motivation can be seen as the force that drives actions. Motivation for 
specific rewards depends on the specific needs of the animal (intrinsic goal functions), 
which can vary from species to species. Intriguingly, humans seem to have intrinsic goal 
functions for social rewards. In addition to the intrinsic goal functions, cues can also 
be learned that can drive motivation, either to specific outcomes (specific Pavlovian- 
to-Instrumental Transfer) or to invigorate the animal in general (general Pavlovian-to- 
Instrumental Transfer). These motivational components are an important part of the 
decision-making system, because they define the goals and how hard you are willing 
to work for those goals, even if they don’t directly select the actions to take to achieve 
those goals. 


Books and papers for further reading 

• David W. Stephens and John R. Krebs (1987). Foraging Theory. Princeton, NJ: 
Princeton University Press. 

• Robert Wright (1995). The Moral Animal: Why We Are the Way We Are: The New 
Science of Evolutionary Psychology. New York: Vintage. 

• Kent C. Berridge (2012). From prediction error to incentive salience: Mesolimbic 
computation of reward motivation. European Journal of Neuroscience, 35, 
1124-1143. 

• Yael Niv, Nathaniel D. Daw, Daphna Joel, and Peter Dayan (2006). Tonic dopamine: 
Opportunity costs and the control of response vigor. Psychopharmacology, 191, 
507-520. 
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The Tradeoff Between Exploration 
and Exploitation 


Waterfall at the edge of the mountains 


beyond the swirling chaos, 
the river vanishes into the dark forest, 
racing to the sea.... 


To take the correct actions within an environment, one has to know 
the structure of the environment, the consequences of those actions. 
Exploration is the process through which a novel environment 
becomes familiar. Exploitation is the process of using that knowledge 
to get the most out of a familiar environment. There is a tradeoff 
between these two processes—exploration can be dangerous, but not 
knowing if there are better options available can be costly. 


An issue that often comes up in theoretical discussions of decision-making is that there 
is a tradeoff between learning more about the world and using what you’ve learned 
to make the best decisions. Imagine that you’re in a casino with a lot of different slot 
machines available to play. You don’t know what the payout probabilities are for any 
of them; to find out, you have to play them. As you play a machine, you learn about its 
parameters (how likely it is to pay how much given its costs). But you don’t know if 
one of the other machines would be a better choice. One can imagine where one would 
fall on this continuum of exploring to see what the payoffs of the various machines are 
and exploiting the knowledge you already have to choose the best option that you know 
of. A This is a task called the n-armed bandit task 1 (as an extension of the original slot 
machine, known as the “one-armed bandit” because it stole all your money). 

In a stable world, it is best to do some exploring early and to diminish your explora¬ 
tion with time as you learn about the world. 2 A good system often used in the robotics 


A The use of “exploit” here has unfortunate overtones in terms of “exploiting others.” What is meant 
here by “exploit” is simply using the knowledge one has learned from exploring to make the best choices. 
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and machine-learning® literature is to give yourself a chance of trying a random machine 
each round. This random level is called e (epsilon) and the strategy is called an “epsilon 
strategy.” The question, then, is what should epsilon be on each round? Simple strategies 
with a constant, low epsilon are called “epsilon-greedy.” For example, you might try a 
random action one round in a hundred and use your knowledge to pick the best choice 
on the other ninety-nine tries out of the hundreds In a stable world, you should explore 
early and then settle down to the best choice later. In the machine-learning literature, a 
common strategy for a stable world is to use an exponentially decreasing chance of try¬ 
ing a random action. This explores early and exploits after you have learned the structure 
of the world you are living in, but never completely stops exploring. 

Of course, the real world is not generally stable, and it would not generally be safe 
to have a pure exploration phase followed by a pure exploitation phase. More recent 
machine-learning algorithms have been based on explicitly tracking the uncertainty in 
each option. 6 When the uncertainty becomes too large relative to how much you expect 
to win from the option you are currently playing, it’s time to try exploring again. This 
requires keeping track of both the uncertainty and the expected value of an option. As we 
will see below, there is now behavioral evidence that people keep track of uncertainty, 
that uncertainty is represented within the brain, and that genetic differences affect how 
sensitive individuals are to uncertainty (how willing they are to switch from exploita¬ 
tion to exploration modes). 7 

The expected value of an option is how much you expect to win on average if you 
keep playing the option; uncertainty is how unsure you are about that expected value. 
This is often called the “risk” in financial markets. 8 Imagine that you know that one time 
in twenty when you put a dollar into the game you get twenty-one dollars out. That 
game then has a positive payout of $21/$20 or 1.05, meaning you make 5% playing that 
game. D On the other hand, another machine might pay $105 one time in 100, which 


B Machine learning is a branch of computer science in which algorithms are developed to learn from 
the world. 3 Although machine learning often lives in the artificial intelligence divisions of computer 
science departments, machine learning is actually distinct from artificial intelligence. “Artificial intel¬ 
ligence” is about trying to match (either mechanistically or computationally) human skills. “Machine 
learning” is about trying to build the best algorithm that will be able to solve the problem. Scientists 
working on machine learning believe that a machine that can learn from the world can he more efficient 
and perform better than a preprogrammed machine. (For example, imagine if your thermostat could 
learn when you typically came home or when you typically woke up in the morning. It could learn 
to set itself to warm the house before you arrived or just before you woke up. Of course, it could be 
programmed to do that, but then it would have to be programmed separately for each person. A single 
learning algorithm could be programmed once to learn the right timing for each person. 4 ) 

c This can he most easily implemented by selecting a random number from 1 to 100 each try. If the 
random number is 100, then you take a random action. If it’s anything else, you pick the best choice you 
can. One could also simply pick the random action every 100th try, hut the danger is that if the winning 
probabilities have some oscillation to it, then your every 100th try might match the oscillation and you 
wouldn’t know. Randomness is better. Of course, humans are terrible at being random. 5 

D Yes, I know slot machines always have a less than 100% payout because the goal of casinos is to 
make a profit. If you like, you can think of these games as putting money into savings accounts. A savings 
account in a bank may be FDIC insured and thus very safe (because the only way to lose money is for the 
United States itself to go under), but you’d make only 1% interest each year. On the other hand, a mutual 

( continued ) 
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also has a positive payout of $105/$100 or 1.05 or 5%, but is much more variable. If 
you played either game for a very long time, the expected value of playing would be the 
same. But if you play each game only a few times, you could lose more (or gain more) in 
the second game. The second one is riskier. 


Uncertainty 

It is useful to think of three different kinds of uncertainty: expected uncertainty, unex¬ 
pected uncertainty, and ambiguity. 10 

In the first type (expected uncertainty), one recognizes that there is a known unpre¬ 
dictability in the world. We see this when a meteorologist says that there is a “60% chance 
of rain.” If the meteorologist is correct, then 6 times out of 10 that the meteorologist says 
“6096,” there should be rain and 4 times out of 10, it should be dry. 

The second type (unexpected uncertainty) entails a recognition that the probabilities 
have changed. This is what is tested in reversal learning tasks, in which reward-delivery 
probabilities are changed suddenly without an explicit cue telling the subject that the 
probabilities have changed. Rats, monkeys, and humans can all reliably track such 
changes. 11 Since the reversal is uncued, recognition of the change in probabilities 
becomes an information-theory problem. 12 Humans and other animals can switch as 
quickly as information theory says anything could. When faced with a sudden change in 
reward-delivery probability (such as in extinction, when rewards are suddenly no lon¬ 
ger delivered, see Chapter 4), animals show frustration, increasing their exploration of 
alternative options. 13 

The difference between expected and unexpected uncertainty was most famously 
(infamously) laid out by Secretary of Defense Donald Rumsfeld in his press conference 
defending the failures of the American invasion of Iraq, in which he claimed that there 
were “known unknowns” (expected uncertainty) and “unknown unknowns” (unex¬ 
pected uncertainty) and that he should not be blamed for misjudging the “unknown 
unknowns.” 14 Of course, the complaint was not that there was unexpected uncertainty, 
but that Rumsfeld and his team had not planned for unexpected uncertainty (which any 
military planner should do ls ) and that Rumsfeld and his team should have expected that 
there would be unexpected uncertainty. 

The third type of uncertainty is ambiguity, sometimes called estimation uncertainty, 16 
uncertainty due to a known lack of exploration. This is what is often referred to in the 
animal-learning literature as the transformation from novel to familiar and is the main 
purpose of exploration. 17 Here, the uncertainty is due to having only a limited number 
of samples. The more samples you take, the better an estimate you can get. Exploration 
entails taking more samples, thus getting a better estimate of the expected uncertainty. 

Humans find ambiguity very aversive. The classic example of ambiguity is to imag¬ 
ine two jars containing marbles and having to decide which jar to pick from. 18 Each jar 
contains a mix of red and blue marbles. You will win $5 if you pick a red marble and 


fund account might not be FDIC insured (and thus you can lose money on it), hut you might make an 
average of 5% interest each year. Buying a stock might be very variable, making 10% one year and losing 
10% the next. This is why good financial planning requires a diverse portfolio to balance out risk. 9 
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$1 if you pick a blue marble. Imagine that you see the first jar being filled, so that you 
know that half of the marbles are red and half are blue. (You will still have to pick with 
your eyes closed and the jar above your head so which marble you pick will be random, 
but you know the distribution of marbles in the first jar.) Imagine, however, that the 
second jar was filled while you were not there and all you know is that there is a mix 
of red and blue marbles. Which jar will you pick from? In the first one, you know that 
you have a 50:50 chance of getting red or blue. In the second one, you might have a 
90:10 chance or a 10:90 chance. Adding up all the possible chances, it turns out that 
you also have a 50:50 chance of getting red or blue. (The likelihood of seeing a 90:10 
mix of red and blue marbles is the same likelihood of seeing a 10:90 mix. Since 90:10 is 
the opposite of 10:90, they average out to being equivalent to a 50:50 mix in both the 
first-order average [expected value] and the second-order variability [risk].) However, 
people reliably prefer the first jar, even when the fact that the second jar adds up to the 
same chance is pointed out to them. The first jar contains risk (expected uncertainty; 
you could get red or blue). The second jar, however, also contains ambiguity ( What’s 
the mix?) as well as risk ( What’s the variability?). This is known as ambiguity aversion 
and is technically an example of the Ellsberg paradox.® 

There are a number of explanations that have been suggested for the Ellsberg paradox, 
including that humans® are averse to ambiguity due to a consequence of being more sensi¬ 
tive to losses than to gains or to an inability to do the full calculation of probabilities needed 
for the ambiguity case. Alternatively, the Ellsberg paradox could arise because humans 
have inherent heuristics aimed at avoiding deceit. 21 Since the subject doesn’t know the 
mix of the second jar, it might have been chosen to be particularly bad, just so the experi¬ 
menter wouldn’t have to pay much out to the subject. (Rules on human-subjects studies 
generally do not allow an experimenter to lie to a subject, so when the experimenter tells 
the subject that there is a 50:50 mix, it has to be true, but when the experimenter tells the 
subject that there is just a “mix,” it could have been chosen to be particularly poor. G But, 
of course, it is not clear that the subjects of these experiments believe that.) 

Animals will reliably expend effort to reduce uncertainty. (This means that knowl¬ 
edge really is rewarding for its own sake.) People often say that figuring something out 
gives them a strong positive feeling, both of accomplishment and of knowing how things 
work. 23 (Certainly, I can say personally that there is no feeling like the moment when a 
complex problem suddenly comes clear with a solution, or the moment when I realize 
how a complex set of experiments that seem to be incompatible with each other can be 
unified in a single theory.) Not only humans, but also other animals will expend energy 
simply to reduce ambiguity. Ethan Bromberg-Martin and Okehide Hikosaka tested this 


E The paradox was introduced by Daniel Ellsberg (of the famous Pentagon Papers 19 ) in his 1962 
Ph.D. dissertation in economics. 20 It is called a paradox because the simple equations of economists are 
unable to explain why humans are ambiguity averse. 

1 To my knowledge, although the paradox is robustly shown by humans, the paradox has not been 
tested on nonlinguistic animals, probably because it depends on the experimenter telling the subject 
the two probabilities. 

G These rules were put in place due to egregious experiments such as the Tuskegee syphilis experi¬ 
ment, where subjects were lied to about the goals and consequences of experiments. 22 These rules are now 
followed very strictly, and it is exceptionally difficult to get approval to lie to an experimental subject. 
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in monkeys and found that information that reduces uncertainty even if it has no effect 
on the final amount of reward the animal gets, has gotten, or expects to get, produces 
firing in dopamine neurons 24 (which, if you remember, we saw represent things being 
“better than expected,” Chapter 4). 


Gathering information 

Computational theories have proposed that a good way to encourage a computational 
agent or robot to explore is to provide “exploration bonuses,” where the system pro¬ 
vides a “better than expected” signal when the agent enters a new part of the world. 25 
Computationally, if this is matched by a “worse than expected” signal when the agent 
returns to its familiar zone, then the total value function will still be learned correctly. 
But it is not clear if the counteracting “worse than expected” signal appears on return. 
People and animals returning to a safe and familiar zone tend to show relief more than 
an “oh, well, that’s done” disappointment signal. 26 Although it is possible to prove stabil¬ 
ity only if an agent is allowed an infinite time to live in a stable world (obviously not a 
useful description of a real animal in a real world), simulations have shown that for many 
simple simulated worlds, not having a “worse than expected” signal on return does not 
disrupt the value function too much. These simulations have found that an agent with 
only a positive “better than expected” novelty signal does as well or better than agents 
with no novelty signals. 27 Such an agent would be overly optimistic about future pos¬ 
sibilities. This idea that the agent should receive a bonus for exploration is reflected in 
the human belief that things could always be better elsewhere. As Tali Sharot discusses 
in her book The Optimism Bias, the data are pretty strong that humans are optimistic 
in just such a manner—“the grass is always greener on the other side.” Humans tend 
to overestimate the expectation of positive rewards in novel choices, which leads to 
exploration. 28 

Several studies have found these sorts of “novelty bonuses” in dopamine signals in 
monkeys and in fMRI activity from dopamine-projecting areas in humans. 29 Animals 
do find novelty rewarding, and will naturally go to explore a novel item placed in their 
environment. 30 This is so natural that it doesn’t require any training at all. Harry Harlow, 
in his early monkey experiments, found that fully fed monkeys would work harder to see 
novel objects than hungry monkeys would work for food. Hungry rats will explore a new 
environment before eating. This suggests that exploration (the search for knowledge or 
information) is an intrinsic goal of the sort discussed in Chapter 13. 


Learning from others 

In the real world (outside of the laboratory), such exploration can, of course, also be 
dangerous. An advantage humans have is that they can imagine futures (episodic future 
thinking, Chapter 9) and can explore possibilities by imagining those potential future 
choices, even without necessarily trying them. One way this can occur is with fictive 
learning, in which one learns from watching others. One can observe another person’s 
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successes and failures and learn about what happens to him or her without directly 
experiencing it oneself. 31 

Of course, this is one of the most powerful aspects of human language, because lan¬ 
guage can be used to inform and to teach. A number of authors have observed that human 
culture is fundamentally Lamarckian; parents really do pass the lessons they learn in their 
lives on to their children. 32 (Although those lessons don’t always stick...) Nevertheless, 
this Lamarckian nature of culture is one of the reasons that humans have spread so 
quickly to all of the corners of the globe, from the hottest jungles to the coldest tundras, 
even sending men to the moon and robots to the other planets in the solar system. 

We now know that when a monkey observes another monkey doing a task that it is 
familiar with, the set of cortical neurons that normally fire when the observer is doing a 
task also fire when the observer is just watching the other monkey do the task. 33 These 
neurons are called “mirror neurons” because they mirror the behavior that the other 
monkey is doing. A similar set of human cortical circuits is active (as determined by 
fMRl) whether one is doing the action oneself or watching it. 34 Although the conclu¬ 
sions of some of these results are controversial ( Are the mirror neurons actually critical to 
imitation, or are they only recognizing related cues? Are they simply reflecting social interac¬ 
tions?), certainly, we all know that we learn a lot by observing others. 35 

Similarly, we also know that we learn vicariously about reward and failure by watch¬ 
ing others. A similar mirror system seems to exist for emotions, where we recognize 
and interpret the emotions of others. 36;H This recognition is related to the emotions 
of empathy, envy, and regret. 39 These emotions are dependent on an interaction 
between hippocampal-based abilities to imagine oneself in another situation and 
orbitofrontal-based abilities to connect emotion to those situations. 40 In empathy, 
we recognize another’s success or loss and feel a similar emotion. In envy, we recog¬ 
nize another’s gain and wish we had achieved it. There is an interesting counter-side 
to envy, gloating or schadenfreude, meaning taking joy in another’s adversity (from 
the German words schaden, harm, and freude, joy), which also depends on the same 
emotion-mirroring brain structures that envy does. And, in regret, we recognize a path 
we could have chosen but did not. 

In large part, this is the key to the power of narrative and literature. We identify 
great stories as those that place realistic characters in situations, watching how they 
react. Great literature allows us to vicariously experience choices and situations that 
we might never experience. If one looks at the great works of literature, whether clas¬ 
sic or popular, * 1 one finds that they consistently describe human decision-making from 
remarkably realistic and complex perspectives. This is true whether one is talking about 


H There is some evidence that part of the problem with psychopaths may be due to dysfunction in 
this mirror system and its ability to recognize the emotions of others, 37 but others have argued that this 
dysfunction is due primarily to an inability to correctly represent socially derived internal emotions. 38 

1 One probably should not make a distinction between “classic” and “popular.” Most literature that 
we call “classic” now was immensely popular in its time. Shakespeare, for example, was by far the most 
popular playwright of his day, and his plays are full of rough jokes for the popular crowds. (See, for 
example, Mercutio’s death scene in Romeo and Juliet, most of A Midsummer Night's Dream, and the 
porter in Macbeth, to name but three.) I’ve often thought Shakespeare at his time was probably clos¬ 
est to Steven Spielberg in our day, with both blockbuster adventures like Indiana Jones and critically 

( continued ) 
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realistic dramas, magical worlds, or science fiction; even science fiction and magical 
realism depend on characters behaving realistically in strange situations. Although read¬ 
ers are generally willing to accept strange situations in other galaxies, or characters with 
magical powers, readers are generally unwilling to work through unrealistic characters.- * 1 
Reading or watching these narratives, we place ourselves in those worlds and imagine 
what we would do in those situations, and we empathize with the characters and feel 
envy and regret about the decisions they make. 


Exploration and exploitation over the life cycle 

Computational work from the machine-learning literature has shown that in an 
unchanging world, the optimal solution to the exploration/exploitation tradeoff is to 
explore early and exploit late. 42 Of course, “early” and “late” have to be defined by the 
agents expected lifetime. This would predict that one would see juveniles exploring and 
adults exploiting. While this is, in general, true, there are other important differences 
that change how the exploration/exploitation balance changes over the lifetime. 

First, as we’ve discussed elsewhere, we don’t live in an unchanging world. Thus, one 
cannot explore early and be done. Instead, one needs to titrate one’s exploration based 
on how well one is doing overall. 43 When the rewards one is receiving start to decrease, 
one should increase one’s exploration. When faced with a decrease in delivered rewards, 
animals get stressed and frustrated and show an increase in a number of behaviors, both 
the behavior that used to drive rewards, as well as other exploratory behaviors, start¬ 
ing with behaviors similar to the one that used to drive reward delivery. We’ve all seen 
ourselves do this, starting with trying the behavior again ( This used to work!), moving to 
minor changes ( Why doesn’t this work?), and then on to new behaviors (Maybe this will 
work instead.). 

Second, exploration is dangerous. This means that evolution will prefer creatures 
that develop basic abilities before going exploring. In humans, young children explore 
within very strict confines set by parents, and those confines expand as they age. As every 
parent knows, kids want to know that there are limits. They want to test those limits, but 
they also want those limits to exist. This suggests that the primary exploratory time in 
humans should occur after the person develops sufficient talents to allow handling the 
dangers of exploring the world. 

The teen, adolescent, and early college years are notorious for high levels of drug 
use, exploratory sex, and risk-taking in general. 44 These are the exploratory years, where 
humans are determining what the shape of the world is outside the confines of their 


acclaimed movies like Schindler's List. Similarly, the entire city of Athens turned out for the annual 
shows by Aeschylus, Sophocles, and Euripides, each of whom regularly won the best-play awards. 
Aristophanes’ comedies contain what we now refer to as “frat-boy ” humor, with lots of sex (Lysistrata ), 
over-the-top characters (The Clouds), and fart jokes (The Frogs). 

1 An interesting question arises when the characters are explicitly nonhuman, whether they be 
aliens, magical creatures, or robots. In these stories, the nonhuman creatures tend either to be based 
on specific human cultures or to have exaggerated but still remarkably human, traits. 41 Writing science 
fiction about aliens can be said to be about finding the balance between making the aliens too human 
and making them too alien. 
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family structure. In general, most humans age out of these vices and settle down as they 
shift from an exploration to exploitation strategy. 

In the next chapter, we will see how the prefrontal cortex plays a role in measuring risk 
and controlling dangerous exploration. One of the most interesting results discovered 
over the past few decades is that the prefrontal cortex in humans doesn’t fully develop 
until the mid-twenties. 45 This means that adolescents do not have the same prefrontal 
assessment of risk and behavioral inhibition of doing stupid things as adults. As we’ve 
seen throughout the rest of this book, behaviors are generated by the physical brain, and 
changes in behavior are often caused by physical changes in the brain. K 

Books and papers for further reading 

• Tali Sharot (2011). The Optimism Bias: A Tour of the Irrationally Positive Brain. 
New York: Pantheon. 

• Abram Amsel (1992). Frustration Theory. Cambridge, UK: Cambridge University- 
Press. 

• B. J. Casey, Sarah Getz, and Adriana Galvan (2008). The adolescent brain. 
Developmental Review, 28, 62-77. 


K As I’ve said elsewhere in this book (see Chapter 24 for a detailed discussion), the recognition that 
there are physical and evolutionary causes for increased risk-taking in adolescents does not excuse the 
adolescent from the responsibility for his or her actions. It is important to recognize that sociobiologi- 
cal trends are not destiny and do not relieve one of responsibility. Although most drug users start using 
drugs in their adolescent years and many drug users age out of drug use and settle down to normal (non¬ 
drug-based) lives, the vast majority of adolescents do not succumb to peer pressure and drug use. 46 One 
hopes that before being sent off to college or out into the world for their exploration, adolescents have 
been taught what their limits are and have an accurate sense of the risks and dangers. 
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Self-Control 


a stone is a mirror 
without concentration 
light reflects in all directions 


We have the conscious experience that we can sometimes "over¬ 
ride” emotional or habitual responses. This means that some pro¬ 
cess must mediate between the multiple decision-making systems. 
The mechanisms of that “override" process are still being studied, 
but some neural components, such as the prefrontal cortex, are 
known to be involved. 


As we’ve seen in the past several chapters, the human decision-making system contains 
multiple action-selection components. Reflexes are largely driven by spinal control, but 
reflexes can be overridden by other systems in the brain (as we saw with Lawrence of 
Arabia holding the match). Emotional (Pavlovian) response systems release behaviors 
learned over evolutionary timescales, but, again, they can be overridden by other systems 
(controlling your fear). Procedural learning systems are habits that develop in response 
to consistent reward contingencies, but again, these can be overridden by other systems 
(as in the case of remembering to take the alternate route to work). And finally, there 
is the Deliberative system, which searches through future possibilities but does not (in 
general) have to pick the most desired choice. (Take, for example, the ability to select a 
small token of reward over a highly craved drug; the craving for a high-value option can 
be overridden to take what seems on the surface to be a lesser-value option.) 

The word “override” comes from the ability of the nobility to dominate decisions 
(since the nobility served as cavalry and could literally override the opposition if it came 
to that) and echoes the Platonic concept of two horses leading a chariot—a wild uncon¬ 
trollable (Dionysian) horse and an intellectual, rational, reasoned (Apollonian) horse, 
as well as the Augustinian and Freudian concepts of human reason controlling a wild 
animal past. 1 These concepts echo the popularized concept of a horse and rider, with the 
horse driving the emotional responses and the rider the reasoned responses. 2 
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As anyone who has ridden a horse knows, riding a horse is very different from driv¬ 
ing a car. The horse has internal sensors and will do some of the basic chores for you 
(like not bump into walls), while a car will do whatever you tell it to. Of course, this 
means that sometimes you have to fight the horse (for example, if you need to drive it 
into battle) in a way that you don’t have to convince a car. (However, I am reminded of 
the scene in John Carpenter’s darkly comic movie Dark Star where Lt. Doolittle has to 
convince “smart” Bomb #20 that it really “wants” to launch and blow up the planet.) 
It will be interesting to see how driving a car changes as we introduce more and more 
self-reliant decision systems into them (such as sensory-driven speed control, automatic 
steering, and GPS navigation systems).Jonathan Haidt has popularized this analogy as 
an elephant and a rider rather than the traditional horse and rider, because the elephant 
is a more powerful creature, and no rider is ever completely in control of the elephant. 4 
(Of course, this misses the possibility that the rider and the horse can understand each 
other so well and be in such sync as to be an even more capable team than either one 
individually.) 

In all of these analogies, there is the belief that the “self” has to work to exert control 
over the “other.” As we noted at the very beginning of the book (Chapter l), you are 
both the horse and the rider (or the elephant and the rider). Robert Kurzban, in his new 
book, rejects the concept of the individual self and suggests that “self-control” is better 
understood as conflict between multiple modules. 5 In the language of this book, the 
decision-making system that is you includes the reflexes you inherited, the Pavlovian 
learning that drives you, the habits you have learned, the stories you tell, and the deliber¬ 
ation you do. Nevertheless, self-control is something that feels very real to us as humans. 
Consciously, we know when we successfully exert our self-control, and (afterwards) we 
know when we don’t. Therefore, it is important to understand what this process is. 


Studying self-control in the laboratory 

Before we can take the mechanisms of self-control apart, we need to find a way to make 
a subject show self-control in the laboratory and to measure it. First, we want to be 
quantitative about when people are showing self-control and when they aren’t. This will 
allow us to examine the mechanisms of failures of self-control. Second, we’d like to be 
able to access self-control directly and not have to wait for a person to fall off the wagon. 
Third, we’d like to be able to examine failures of self-control in nondangerous situations. 
Fourth, we’d like to be able to use our big measuring machines (fMRI, EEG, neural 
recordings) to see the physical correlates of self-control. And fifth, we’d like to be able to 
examine self-control in nonhuman animals. 

Self-control has been a topic of study since the inception of modern psychology. 
(Freud’s concept of the id, ego, and superego is a multistage theory of self-control. 6 ) 
Several particularly sensitive measures of self-control have been introduced over the 
years, including the Stroop task and the stop-signal task, as well as tasks that put subjects 
in more realistic situations, such as the marshmallow task and its animal analogs that 
compare abstract and concrete representations (the chimpanzee and the jellybeans). 7 
Finally, there is the candy-rejection task, which will lead us into issues of cognitive load 
and how self-control can tire out. 8 
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Perhaps the simplest experiment to address the question of self-control is the Stroop 
task, a remarkably simple yet subtle task in which a person is shown a word in a color 
and is supposed to name the color of the text. This is normally very easy. But if the word 
is itself a color (such as the word “red” written in blue), then an interference appears 
between reading the word itself and recognizing the color that it is printed in. This inter¬ 
ference produces a slowing of the time it takes to say the word (the reaction time) and 
increases the likelihood of making an error. Both of these measures produce quantitative 
measures of cognitive dissonance and track other measures of self-control. 9 The increase 
and decrease in reaction times in the Stroop task is so reliable that it is now used to test 
for cognitive dissonance and attention to other concepts—for example, for food-related 
words in obese individuals and drug-related words in addicts. 10 

The Stroop task can’t be used with animals (who can’t read), and here the stop-signal 
task has been very useful. In the stop-signal task, the subject is trained to react as quickly 
as possible to a go signal (such as “when the light comes on, push the button”), but 
sometimes a different signal (a Stop! signal) appears between the go signal and the sub¬ 
ject’s reaction. If the subject sees the stop signal, he or she is rewarded for not reacting 
and is punished for reacting—the subject has to stop the already started motor-action 
sequence. The stop-signal task has successfully been used in rats, monkeys, and humans, 
and performance is similar across the different species. 11 

Generally, the most successful description of the task is that there are separate pro¬ 
cesses each racing to a threshold (ago process and a Stop! process). The Stop! signal is 
assumed to run faster than the go signal, but also to start later. If the Stop! signal reaches 
threshold before the go signal, the subject is able to cancel the action and stop himself 
or herself. 12 Neural recordings have found increasing activity in certain areas (cortical 
motor control structures, and the subthalamic nucleus [a part of the basal ganglia]) that 
track the two racing components and their interaction. 13 

Both the Stroop and stop-signal tasks are computer-driven, timed tasks. The 
other tasks listed above depend on physically real objects being offered to subjects; 
this produces different results than symbols or images of objects. 14 In the first few 
chapters of the book, we encountered an experiment (the Parable of the Jellybeans, 
Chapter l) in which chimpanzees had to choose between two sets of jellybeans. 
The key to this task was that the subject got the tray he didn’t pick—to get the tray 
with more jellybeans, he had to reach for the other one. 15 Sarah Boysen and Gary 
Berntson found that their subjects were much better at picking the “humble por¬ 
tion” if they were using symbols rather than actual jellybeans. Humans also show a 
preference for real objects, particularly sensory-positive objects, when faced with 
experimental decision tasks. 16 

This preference for real objects leaves us with an important (and interesting) incon¬ 
sistency in decision-making systems and underlies some of the problems with valuation 
that we saw in our initial discussion of “value” (Chapter 3). The idea of a thing and the 
physical thing may have very different values. This occurs because there are multiple 
decision-making systems competing for that valuation. If the Pavlovian system (Chapter 8 ) 
is engaged (when the object is physically in front of us), we make one valuation, but if it is 
not (if we are using the Deliberative system, Chapter 9), we make another. 

This creates an interesting inconsistency in decision-making. 17 Even though an alco¬ 
holic might deny the urge to drink when sitting at home, that same alcoholic sitting in 
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the bar an hour later will take that drink. Some authors have argued that the way to 
think about this is that there are two selves: that your present self is in conflict with 
your future self. Other authors have suggested that there are two modules, one that 
thinks long term and one that responds more immediately. I think its simpler to rec¬ 
ognize that we are inconsistent and irrational and to figure out how to work our lives 
around that inconsistency and irrationality. One way to exert self-control is to rec¬ 
ognize this inconsistency and to precommit to a condition that does not allow you to 
reach that future difficult choice. 18 For example, the alcoholic might realize that it will 
be very difficult to not drink in the bar and might decide not to go to the bar in the first 
place. A gambler with a known problem might take a different route home so as not to 
drive by the casino. 19 

This suggests the importance of attention and distraction in self-control 20 —if we can 
keep our attention on the other option, we may be better able to avoid the Pavlovian 
choice. Similarly, if we can attend to specific aspects of an alternative, we maybe able to 
change the valuation of that alternative against the temptation. This may be one of the 
reasons that personal religion is often a source of strength in the face of severe danger 
or temptation. Religion may enable people to maintain their attention on an alternative 
option that precludes committing the “sin” that they are trying to control. 21 

But what if you are faced with that immediate choice? Is it a lost cause? What are the 
mechanisms of self-control? Even the alcoholic facing a drink in a bar can sometimes 
deny that driving urge to drink. 

In the classic marshmallow task, introduced by Walter Mischel in the 1960s, 22 a child 
is brought into a room, a marshmallow (or equivalent high-value candy or toy) is placed 
in front of the child, and the child is told, “If you can wait fifteen minutes without eat¬ 
ing the marshmallow, I’ll give you two marshmallows.” Then the adult leaves the room. 
Videos from the original experiments are now famous in the psychological literature 
because the children show classic self-control behaviors. They try to distract themselves 
from the marshmallow. They cover their eyes so they don’t have to look at the marsh¬ 
mallow. They turn away. They do other actions like kicking the desk. The problem is 
that keeping attention off the marshmallow is hard. Follow-up experiments have shown 
that the ability to wait is surprisingly well correlated to later success, including staying 
out of jail, avoiding drug addiction, getting good grades in high school, SAT scores, and 
job success. 

Adults are generally able to reject a single marshmallow in this situation, but reject¬ 
ing that marshmallow takes cognitive effort. This leads us to the candy-rejection taskP 
In this task, young women who have expressed an interest in dieting are asked to watch 
a movie with a bowl of high-sugar snacks placed next to them. By quantitatively mea¬ 
suring how many of the snacks are eaten, one can quantitatively measure their lack of 
self-control. Some of the women are asked to not respond to the movie, while others 
are told to just watch it. The more they have to control their emotions while watch¬ 
ing the movie, the more they eat. A host of similar self-control and fatigue tests can be 
used as the quantitative test, including willingness to drink a bitter-tasting medicine, 
willingness to hold one’s hand in a cold ice-bath, and physical stamina on a lever-pulling 
or handle-squeezing task. All of these require some aspect of self-control, and they are 
impaired (decreased) by preceding them with tasks that engage excess emotional and/ 
or cognitive components. 
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This leads us to the concept of cognitive load. Many of these tasks, such as the 
candy-rejection task, and even the Stroop task, depend on the ability to allocate some 
sort of limited cognitive resource to the self-control component. 24 We can see this most 
clearly in the candy-rejection task—spending your self-control attention and effort on 
keeping your emotions in check reduces your ability to reject temptation and increases 
the amount of candy that you eat. Putting your override abilities elsewhere allows your 
Pavlovian system free rein. (Ask anyone nibbling on chips and pretzels in a sports bar.) 
In fact, we see this sort of effect all the time—when one is distracted, one tends to forget 
to override prepotent actions, whether they are driven by the emotional (Pavlovian) 
system or by the overlearned (Procedural) system, or whether they are highly motivated 
targets in the Deliberative system. 25 This is a specific instantiation of the general process 
of having limited resources. Complex processing, including memory, navigation, and 
self-control, all require a similar set of limited resources. If these resources are taken up 
by one task, they cannot be used in another. 26 

In one of my favorite examples of limited cognitive resources, following on an 
experiment introduced by Ken Cheng and Randy Gallistel for rats, 27 Linda Hermer 
and Elizabeth Spelke tested how well children (two-year-olds) could find a favorite 
hidden toy. 28 The toy was placed in one of two boxes in the corners of a rectangular 
room. The key is that a rectangular room has a 180-degree rotational symmetry. The 
children watched the toy being placed in the box and were then spun around until they 
were dizzy. Although the children had watched the toy being placed in the box only a 
few minutes earlier, they were unable to remember where it was; they were able to use 
the geometry of the room to identify two of the four corners, but they were unable to 
remember which of the two opposite corners had the box. A Even with colored boxes and 
salient wall cues (one wall painted white, one painted black), the children were unable 
to break that 180-degree symmetry. (Children who were not spun had no trouble at all 
finding the favorite toy.) Adults were of course able to remember even through being 
spun. (Of course, what the adults were doing was saying, “It’s left of the black wall” 
or “It’s in the blue box”) If the adults were given a linguistic blocking task at the same 
time (to stop them from repeating the location over and over again, linguistically), they 
ended up just like the children—unable to remember where they left the toy. Classic 
cognitive load tasks used in experiments include saying the alphabet backwards, and 
counting down from 1000 by sevens. (Counting down from 1000 by sevens is really 
hard to do and really distracting.) 

Distracted people revert to the other systems. I recently changed the route I drive 
to work. If I get distracted (say by thinking about how I’m going to deal with a prob¬ 
lem at work or at home or even just by thinking about writing this book instead of 
driving), then I miss my turn and go straight through to the old route. I’m sure if I 
was counting down from 1000 by sevens as I drove through that corner, I would miss 
it every time. 


A Rats were trained to find food. Even with highly salient cues in each corner (different smells, num¬ 
bers of colored lights, etc.), rats who were disoriented each time before being placed in the box were 
unable to break that 180-degree rotational symmetry. 29 Rats who were not disoriented were easily able 
to break the symmetry. See my earlier book Beyond the Cognitive Map for a detailed discussion of the rat 
experiment and what is known about its neurophysiological correlates and mechanisms. 
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The neurophysiology of willpower 

Using these limited resources takes energy. One of the most interesting things about 
self-control is that not only does it depend on paying attention, and not only does it 
diminish when one is tired, but accomplishing self-control takes energy. 30 (Humans are 
aware of this. We talk of “willpower” as if it is a limited resource that we need to conserve 
for when we need it. In fact, if subjects are told that they will need to use their self-control 
later, they can conserve it, showing less self-control in earlier tasks but then being more 
able to express self-control in later tasks.) Even something as simple as resisting tempta¬ 
tion (such as not eating offered chocolate) can have an effect on ones ability to perform 
some physical task requiring stamina. 31 

Many scientists argue that this is one of the reasons that stress often leads to fall¬ 
ing back into habits that have been stopped, such as smoking or drug use. Coping with 
stress depletes ones self-control and leaves one vulnerable to tasks that depend on that 
self-control resource, such as preventing relapse. In fact, it has been argued that one of 
the reasons that regular meetings (such as in Contingency Management or 12-step pro¬ 
grams such as Alcoholics Anonymous and its more-regulated cousins) work so well is 
that they provide daily rewards for accomplishing self-control, which helps strengthen 
those self-control “muscles.” 32 In a similar vein, Walter Mischel has argued that much of 
what we call good parenting entails an explicit training of self-control 33 (such as telling 
kids that “it’s ok to wait for two marshmallows”). 

These self-control resources can be replenished through sleep, rest, or relaxation. 34 
Roy Baumeister and his colleagues have suggested that this limited resource depends 
on glucose levels in the bloodstream; 35 however, this suggestion is extremely contro¬ 
versial. 36 Glucose (sugar) is a resource required by neurons to power neural function. 37 
Blood-glucose levels in subjects performing a self-control task were much lower than in 
subjects performing a similar control task that did not require self-control, and subjects 
who drank a sugar drink showed improved self-control relative to those who drank a 
similarly sweet but artificially flavored drink. 38 Apparently, eating sugar really does help 
with self-control, which maybe why stressed people crave sweets.® If this glucose theory 
is correct, then self-control may really be like a muscle, in that it gets tired because it 
runs out of resources. However, other scientists have suggested that the taste of glucose 
may be priming expectations of goals and changing the distribution of which systems 
are driving decision-making. Merely tasting (but not swallowing) a sweet solution 
can increase exercise performance. 39 Perhaps the glucose solutions are changing ones 
motivation, invigorating the subject (Chapter 13). 

It is also not clear why self-control requires additional glucose compared to other 
brain-intensive phenomena, such as increased attention (listening for a very small 
sound), increased perception (finding a hidden image in a picture), fine motor control 
(playing piano or violin), or even doing a cognitively difficult problem (a math test). 
To my knowledge, the glucose theory has not been tested in these other brain-intensive 


B I know that when I’m writing a grant application or working really hard on a paper, I often find that 
I need a regular infusion of severe sugar to keep my writing stamina up. You can tell I’m working hard 
when I’ve got a donut for my midmorning snack. 
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phenomena. 0 It may be that self-control simply engages more neural systems than just 
going with the flow does, and those neural systems require resources. 

There is also evidence that self-control can be trained, 42 but whether this is more akin 
to a practice effect (through which neural systems learn to perform tasks better) or to 
a homeostatic change (such as muscles increasing their resource buffers from exercise) 
is still unclear. 

So why does self-control engage additional parts of the brain and require more neu¬ 
ronal resources? 


Self-control and the prefrontal cortex 

Two structures in the human brain appear again and again in self-control tasks: the dor¬ 
solateral prefrontal cortex (dlPFC) and the anterior cingulate cortex (ACC). 43 These 
structures are often talked about as being involved in “top-down control” because they 
are involved in abilities to recognize and change goals and tasks. 44 These two structures 
seem to be involved in the monitoring of conflicting information and desires, task-setting, 
and the overriding of plans. 45 Both of these structures are located in the front of the 
human brain, and they are often referred to as part of the “prefrontal” cortex. D 


c As discussed in Appendix B, fMRI measurements depend on increased blood flow because 
“highly active” neurons require additional resources (but not additional oxygen) from the blood. 
Although it is not known what all of those additional resources are, a likely candidate is glucose. 40 
So, when a neural structure increases its activity (whatever that actually means 41 ), the brain accom¬ 
modates those needs by increasing blood flow. However, the increased flow is much larger than the 
increased oxygen needs, so with increased blood flow and the same oxygen needs, more oxygen is 
left in the bloodstream, which is what is detected by fMRI and similar imaging systems. Increased 
activity can also be tracked by using a radioactive glucose molecule, 2-deoxyglucose. The radioactive 
2-deoxyglucose is injected into an animals bloodstream and is then taken up by particularly active 
neurons. The presence of the marked 2-deoxyglucose can be detected postmortem in certain tissues, 
which have been “highly active.” 2-deoxyglucose tends to mark the same structures that fMRI does. 
The fact that fMRI identifies certain structures as “involved” in certain tasks suggests that different 
abilities draw on different parts of the brain and that the glucose hypothesis may affect more systems 
than just the self-control systems. 

D The term “prefrontal” comes from the idea that it is in the “front of the frontal cortex.” Because 
this part of the cortex is one of the most differentiated between primates and other mammals and 
between humans and other primates, 46 it was historically identified as the locus of the abilities deemed 
to be uniquely human (such as the “superego” 47 ). However, as anatomical studies have become more 
detailed and more developed, it has become possible to identify homologous areas in the frontal 
brain regions of other animals as well. 48 (A homologous structure is one that serves the same purpose 
between two species and shares an evolutionary history. For example, no one would deny that the rat 
heart, the monkey heart, and the human heart are all homologous structures. As we have seen through¬ 
out this book, many brain structures are conserved between these species as well.) Similarly, as we 
have begun to develop self-control tasks that can be used in animals (such as the stop-signal task), we 
have begun to identify specific pathways, structures, and information-processing mechanisms through 
which self-control works 49 The exact details of the homology between species and the exact details of 
their respective roles in decision-making are an area of active research. 50 
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The implication that the prefrontal cortex is involved in self-control originally came 
from inadvertent lesion studies, including soldiers returning from the wars of the late 19th 
and early 20th centuries. 51 But the first and most important indication that the prefrontal 
cortex played a role in self-control, particularly the Augustinian nature of it, came from 
Phineas Gage, 52 whom we met in our discussion of emotion (Chapter 8). To remind you, 
Phineas Gage was a railway worker who had an iron spike shot through his head, obliter¬ 
ating his frontal cortices. One of the key descriptions of Phineas Gage after his accident 
was his lack of self-control, particularly in emotional situations. More modern results, 
including both fMRI data from humans and neural recordings in monkeys, have found 
dorsolateral prefrontal cortical activity to be related to working memory, to self-control, 
and to the construction and maintenance of alternate plans (such as one might take in the 
face of an error-related contingency), particularly complex alternate plans. 53 

Because the stop-signal task is trainable in animals, particularly monkeys, the ana¬ 
tomical pathways through which the “stop” behavior is achieved are well known. In a pair 
of remarkable recent papers, Masaki Isoda and Okehide Hikosaka found that an area to 
the front of the motor cortex (the “supplementary motor area” [SMA]) stops unwanted 
behaviors through its strong projection to the subthalamic nucleus in the basal ganglia. 54 
In our discussion of the Procedural action-selection system (Chapter 10), we talked 
of go/no-go pathways in the basal ganglia—cortical input enters the striatum in the 
basal ganglia and is then passed through two pathways (a “go” pathway, often called the 
“direct” pathway, in which the striatum inhibits an area that inhibits actions, making a 
double negative that learns to encourage actions, and a “no-go” pathway, often called the 
“indirect” pathway, in which there is a triple negative that learns to discourage actions). 
There is also a third pathway through the basal ganglia in which the frontal areas of 
the cortex project directly to the subthalamic nucleus, which projects directly to the 
final negative stage of the two pathways. (See Figure 10.1.) The subthalamic nucleus is 
excitatory (positive) and excites the final inhibitory stage, making a single negative— 
a Stop! pathway. To differentiate it from the direct “go” pathway and the indirect “no-go” 
pathway, it is sometimes called the “hyper-direct” pathway. 55 

This can be seen as a general phenomenon—the frontal cortices provide signals that 
allow other structures to change previously learned responses. 56 Historically, this has 
been called the behavioral inhibition system because it is usually tested in experiments in 
which behaviors are stopped rather than changed, 57 but I’ve always suspected that the 
prefrontal cortex is better understood as a biasing system that allows complex plans to 
control behavior through the manipulation of simpler systems. This architecture reflects 
the subsumption architecture proposed by Rodney Brooks 58 in which more complex sys¬ 
tems are overlaid on top of simpler systems. The more complex systems listen to the 
inputs and step in if needed by either modulating the simpler systems (by providing 
them with additional inputs) or by directly driving the output themselves. 

In the rodent, the anterior cingulate cortex (ACC) and the medial frontal cortex 
(particularly the infralimbic and prelimbic cortices) provide signals to the amygdala 
to override learned fear-related responses. 59 In the monkey, the prefrontal cortex over¬ 
rides habitual responses to allow more controlled responses. In general, this can be seen 
as a form of what is sometimes called top-down processing, 60 in which frontal cortical 
systems modulate the “lower” systems to change expectations, attention, and behav¬ 
iors. Reflexes may be driven by the spinal cord, but there is a loop through which the 
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brain examines the inputs to the reflexes and can modulate them if needed. The ACC 
modulates emotional action-selection systems. 61 The frontal motor areas (like the 
SMA) modulate Procedural learning. 62 And dorsolateral prefrontal cortex modulates 
evaluation systems to select longer-term options in Deliberative systems. 63 

To see how this works, we can take a specific example. In a recent fMRI study Yadin 
Dudai and his colleagues in Israel studied humans overcoming their fear of snakes. 64 
They found people who had specific phobias of snakes and gave them the chance to 
pull a rope that brought a snake closer to them. The sight of the snake produced activ¬ 
ity in the amygdala and limbic emotional areas, which have long been associated with 
fear in both animals and humans. (We can recognize these as Pavlovian [emotional] 
action-selection circuits, Chapter 8.) People who were able to overcome their fear (pull¬ 
ing the rope) had decreased activity in the amygdala, but increased activity in the sub- 
genual anterior cingulate cortex (sgACC), an area of prefrontal cortex that projects to 
the amygdala. Somehow, activity in the sgACC has inhibited activity in the amygdala 
and allowed these people to overcome their fear and pull the rope toward them. 

Books and papers for further reading 

• Roy F. Baumeister, Todd F. Heatherton, and Dianne M. Tice (1994). Losing Control: 
How and Why People Fail at Self-Regulation. San Diego, CA: Academic Press. 

• George Ainslie (2001). Breakdown of Will. Cambridge, UK: Cambridge University 
Press. 

• Robert Kurzban (2010). Why Everyone (Else) is a Hypocrite. Princeton, NJ: Princeton 
University Press. 

• Uri Nili, Hagar Goldberg, Abraham Weizman, and Yadin Dudai (2010). Fear Thou 
Not: Activity of frontal and temporal circuits in moments of real-life courag e. Neuron, 
66, 949-962. 


This page intentionally left blank 



PART THREE 


THE BRAIN WITH A MIND 
OF ITS OWN 






This page intentionally left blank 



16 

The Physical Mind 


Between two mirrors, 
light reaches to infinity, 
seeing 

requires an embodied interruption. 


The mind is physically instantiated by the brain, but the mind is not 
simply software running on the hardware of the brain. The physical 
structure of the brain changes the computations that are available. 
We saw earlier that there are multiple action-selection systems, each 
with different advantages and disadvantages. Although it is pos¬ 
sible to learn new tasks after brain lesions, it is not that the same 
software is running on another part of the brain; instead, other 
action-selection systems are doing their best to accomplish tasks. 


Historically, debates about the relationship between mind and brain have been between 
those who argued that they were fundamentally different things ( dualism ) and those 
who argued that the mind was instantiated in the brain. 1 There is strong evidence against 
dualism because manipulations of the brain affect the mind. There is no evidence for 
a nonphysical entity, and few scientists still consider it a possibility. However, more 
recently, duahsm debates have been between whether the mind depends on the physi¬ 
cal brain, or whether it is a software that just happens to run on neural hardware. 2 

We now have examples of things that can be physically translated from one instan¬ 
tiation to another without any loss. When we download a song from the Internet, the 
music has been translated from sound waves in the air through a microphone into a 
digital representation on a computer hard drive, translated from that onto a series of 
electrical pulses over a wire, often through an intermediate step of light pulses over 
a fiber-optic cable, then back into electrical pulses, back into a digital representation 
on your local hard drive, and then back into sound waves in the air through speakers 
attached to your computer. Although there is some change from the sound waves to 
the digital representation and back, there is no change at all from one digital repre¬ 
sentation to another. The physical nature of the mind is not like that. To understand 
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how the material nature of how the brain processes information is different from the 
immaterial nature of digital information, let us take a moment to understand just what 
digital information is. 


What’s wrong with the software/hardware analogy? 

Within a computer, information can be transferred from one place in memory to another 
(as when you copy a file from disk to memory) or, within a network, from one com¬ 
puter to another. The information may be stored in physically different mechanisms, 
but the information itself is the same. For example, digital information stored in the 
form of magnetic orientations on a hard drive can be written to a compact disc (CD) in 
the form of dark spots on an optical disk. A This has led computer scientists to separate 
the concept of hardware (the physical entity) from the concept of software (the digital 
information). 

There are cases in biology where information is stored digitally. For example, DNA 
stores the genetic sequence as a series of four amino acids (guanine, cytosine, thymine, 
and adenine, usually labeled GCTA). One can make a case that the information in DNA 
is software and the cells complex mechanisms that turn that information into proteins 
and other physical changes in the cell are the hardware that reads and operates on the 
software. 6 

In the 1970s and 1980s, the hardware/software analogy became popular, both 
among scientists and the public, as an explanation for the mind/brain problem, with 
the idea that the brain was the hardware and the mind was the software. 4 This software/ 
hardware dualism was particularly appealing because it meant that the mind occurred at 
a different level than the brain and could be studied separately. It also held out the hope 


A A hard drive (HD) stores information as a series of small pieces of magnetizable materials called 
ferromagnetics. Under the influence of a magnetic field, these small spots (called domains) can be 
oriented in one of two directions (up or down). The hard drive stores a series of zeros and ones as a 
sequence of up and down domains. A compact disc (CD) stores information as a series of small pits in 
a colored plastic, making a series of dark and light spots. Hard drives and CDs can store the exact same 
series of zeros and ones. Although instantiated in the physical realm (magnetic domains, dark/light 
pits), the actual information is the sequence of zeros and ones. 

B The actual mechanism by which a cell reads the information stored in DNA is extremely complex, 
with new mechanisms discovered over the past several decades where the physical environment within 
and around the cell changes which components of the DNA are read. 3 For example, specialized pro¬ 
teins called histones can block part of the DNA from being read, while other proteins called transcription 
factors can encourage part of the DNA to be read. At this point, geneticists have created entire logic 
control structures (if this protein is present but that protein is not, then read this part of the DNA to 
create a third protein). The physical environment of the cell can change which part of the program is 
read. But this is not that different from current computer programs, which have mechanisms to check 
physical aspects of the computer to decide what parts of the program need to be run. For example, 
when your computer starts up, it checks to see if you have a DVD reader installed or a network card. 
(These days, programs don’t tend to tell users what they are doing, so it may just show you a little blue 
bar cycling around or a corporate logo to look at, but what it’s really doing is checking for hardware.) 
If it finds that hardware, it installs appropriate drivers (small pieces of software) for it. 
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that a computer program could be built using a completely different hardware and still 
create an artificial intelligence. 

At the time, the software being studied was generally symbolic—sets of arbitrary 
symbols that were being manipulated through specific algorithms. 5 In 1956, Allen 
Newell and Herb Simon showed that a search algorithm that looked through possi¬ 
bilities of mathematical steps could find basic mathematical proofs, showing that they 
could replicate many of the proofs from Bertrand Russell’s Principia Mathematica. 6 
In fact, their program found a more elegant proof to one of the theorems in Russell’s 
Principia than Russell had. 7 

With faster and larger computers, the search process that was developed by these early 
artificial intelligence researchers has been able to do remarkable things 8 —it is as good 
as or better than human doctors at identifying diseases, c it is used by oil-exploration 
industries to find oil pockets deep in the ground, and, famously, it beat the best chess 
player in the world. Recently, the combination of search processes with a deep knowl¬ 
edge base of semantic relationships beat the best human players at the TV game show 
Jeopardy. However, it is important to remember that this search-through-possibilities 
theory is a hypothesis as to how humans make decisions. Extensive psychological tests 
have shown that humans do not search as many steps into the future as these programs 
do. 10 Better chess players do not look farther ahead than weaker chess players. Doctors 
have to be explicitly trained to use differential diagnosis paradigms, and even well-trained 
doctors tend to use experience-based pattern recognition rather than explicit reasoning 
in practice. 

The search-through-possibilities theory requires a manipulation of symbols. The 
reason artificial intelligence researchers started with symbol manipulation (as told by 
Allen Newell in one of his last public lectures given at Carnegie Mellon 11 ) was that 
scientists thought that “perception was going to be easy,” so they would start with the 
hard part of cognition. At the end of his career, Allen Newell came to the conclusion 
that in fact they had been going at it backwards, that perception was far from easy and 
that cognition was actually perceptions applied to the perception mechanisms. But at 
the time (from the 1950s through the 1980s), artificial intelligence entailed manipula¬ 
tions of symbols and led to what Newell and Simon called “the physical symbol systems 
hypothesis,” which proposed that cognition consisted entirely of the manipulation of 
abstract symbols. 12 

Even in the 1980s, the concept that the mind entailed the manipulation of symbols 
was attacked, primarily by John Searle, a philosopher at the University of California, 
Berkeley. Searle attacked the symbol-manipulation hypothesis with a thought experi¬ 
ment that he called the Chinese room. 13 To understand Searle’s thought experiment, we 
need to first start with the famous Turing test, proposed by Alan Turing in 1950. 14 

How do I know that you are conscious? Turing rephrased this question in terms of 
thinking, as in Can a machine think? As we have seen throughout this book, in order to 
be scientific about this, we need to operationalize this question. Turing suggested that 


c Even though early (1979) artificial intelligence programs outperformed typical medical faculty 
in controlled comparisons, these programs were abandoned due to concerns about legal issues —Who 
would he responsible for errors in diagnosis? A similar concern may be limiting successful automatic 
driving programs. 9 
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we can operationalize the concept of thinking by trying to determine if the person you 
are having a conversation with is a computer or a human. Unfortunately humans are 
actually very bad at this and often attribute human characteristics to nonhuman objects 
(such as naming a GPS). 

Searles Chinese room thought experiment is a direct attack on the concept of the 
Turing test. In Searle’s thought experiment, imagine that a Chinese speaker is hav¬ 
ing an email chat with someone in Chinese." 0 Our speaker types Chinese ideograms 
into the computer and reads Chinese ideograms on the computer. Somewhere else, a 
non-Chinese speaker sits in an enclosed room with a set of index cards. Chinese charac¬ 
ters appear on the screen in front of him, and he matches the ideograms (which he does 
not recognize, but we assume he can match them pictorially) to index cards that have 
complex if-then statements on them. Some of those cards tell him to draw specific ideo¬ 
grams on a tablet, which is then translated to the outside. 0 Searle correctly points out 
that even if the outside Chinese speaker (our user) cannot tell the difference between 
the enclosed room and another Chinese speaker (that is, the room passes the Turing 
test), we cannot say that our inside man speaks Chinese. This is completely correct. 
However, it also completely misses the point of the Turing test. While our inside man 
does not speak Chinese, the room (the man manipulating the index cards, the rules on 
the index cards themselves, and the input-output system) does. 

This is a common mistake made by people trying to understand brain functional¬ 
ity. We keep looking for mechanisms where consciousness resides (in a specific neu¬ 
ron subtype or in the quantum fluctuations of microtubules 16 ), but this is not how the 
brain works. The brain is an interactive, computational machine that takes information 
in from the world, processes that information, and then takes actions to respond to it. 
There are many modules in the brain (we explored some of them in the previous section 
of this book), but, in the end, it is the person as a whole that takes an action. When you 
hold your sleeping baby daughter for the first time, that uncontrollable, overwhelming 
emotion created by your Pavlovian action-selection system is you, not someone else; 
when your Procedural system slams on the brakes of your car and avoids the accident, 
that was you making the right choice. Conscious or not, it s still you. 

There are two important hypotheses underlying the hardware/software distinction. 
The first hypothesis is that there are different levels of description of an object. 17 One 
can describe the digital information stored on a hard disk or DVD in terms of the physi¬ 
cal object that it is stored on (for example, as changes in orientation of the ferromagnetic 
domains on the hard disk) or in terms of the information that is stored (as the sequence 


D At the time that Turing proposed his test, getting input and output from computers was very 
complicated, and Turing spent a large part of his original proposal ensuring that the test can’t he fooled 
by seeing or hearing the person or computer. We are now very familiar with the concept of commu¬ 
nicating with invisible entities over computer networks, and, in fact, the Turing test regularly occurs 
in the context of videogame interactions, where some of the characters are computer-controlled and 
some are human-controlled. 

E Apparently, this was, in fact, how the first version of Newell and Simon’s General Problem Solver 
was implemented. Computers were too difficult to program at the time (in 1956), so they encoded it 
on 3 X 5 index cards and had their families manipulate the cards symbolically by specific rules, thus 
producing, literally, the Chinese room. 15 
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of zeros and ones). The second hypothesis is that the information level is independent 
of the physical description 18 (that is, the sequence of zeros and ones stored on the hard 
disk is the same as the sequence of zeros and ones stored on the DVD). Both of these 
statements are incorrect when applied to the physical brain. While we will find both 
the physical and computational or informational levels useful in understanding brain 
function, experimental evidence suggests that the hypotheses that these are distinct and 
separable descriptions does not apply to the brainT 

The hardware/software analogy arose from two scientific literatures (an inability to 
explain consciousness from physical effects 20 and observations of patients recovering 
from lesion studies 21 ) and gained traction as artificial intelligence researchers tried to 
show that symbol manipulation could explain intelligence. 22 Psychology experiments 
have found that symbol manipulation is particularly hard for humans, and that the more 
expertise one has with a subject, the less symbol manipulation one seems to do. 23 This 
suggests that there is something else going on beyond symbol manipulation. 

It is tempting to throw up our hands and say that because we can’t yet explain how 
consciousness arises from the brain, it can’t be a physical object. However, physical 
manipulations affect conscious thought, 24 and operationalizing consciousness has been 
elusive. 2S Every time scientists operationalize a part of consciousness so that it can be 
studied experimentally (for example, by concentrating on one aspect such as attention, 
perception, emotion, or voluntary motor control), these studies have produced detailed 
physical (and computational) explanations for how the process works in brain tissue. 26 
Similarly, when we actually test the timing of consciousness, much of it seems to occur 
after rather than before behavior. 27 Not having an explanation for consciousness seems 
to be a rather weak starting point for the major theory that consciousness is the software 
running on the brain’s hardware. 

Historically, however, what really seemed to drive the software analogy was the obser¬ 
vations that animals and patients could sometimes recover from brain injury. 28 Patients 
with aphasia (language impairments) due to damage to part of their brain would some¬ 
times (with sufficient training) recover some language abilities. 29 Animals with brain 
lesions would sometimes be able to learn to do tasks that they couldn’t do shortly after 
the lesion. 30 And, most importantly, long-term memories seemed to be stored in a dif¬ 
ferent part of the brain from recently stored memories. 31 


Learning new tasks with different brain structures 

The idea that the brain is an amorphous container and that all parts of the brain are 
equivalent goes back to a famous experiment by Karl Lashley in the 1920s in which he 
observed rats running mazes with large swaths of brain tissue removed. 32 Lashley cer¬ 
tainly recognized that there was a difference between parts of cortex (for example, visual 
cortex was obviously specialized for vision, even in the rat 33 ), but Lashley found that the 
ability of rats to solve a maze depended on the amount of brain tissue removed—the 


1 The symbol-manipulation hypothesis is a hypothesis about how we process information. 19 It 
is wrong—not because it cannot be true, but because it doesn’t seem to be how the brain processes 
information. 
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more tissue he removed, the worse the animal did. Generally, Lashley didn’t see much 
difference between lesions that were of similar size but that involved different parts of 
the cortex—but he also wasn’t watching the rats do much in the first place. When we 
actually observe not just whether an animal accomplishes a task, but how the animal 
accomplishes the task, what we find is that animals (including humans) bypass brain 
lesions not by moving information but by solving tasks using other systems. 

We have already seen a difference between Deliberative decision-making (Chapter 9) 
and Procedural decision-making (Chapter 10). This distinction matches very closely 
a distinction seen in the human psychological literature between declarative and pro¬ 
cedural memory. 34 Although the idea that there were multiple, dissociable memory 
systems dates back to the 1970s, the specific distinction between declarative and proce¬ 
dural memory in human learning, and the specific idea that they were dissociable, goes 
back to a 1980 paper by Neal Cohen and Larry Squire and the famous patient H.M. 35 

H.M. (now known to be Henry Molaison G ) was a young man who suffered terri¬ 
bly from intractable, debilitating epileptic seizures that made it impossible for him 
to work. 37 Epilepsy is a breakdown in the negative feedback systems in the brain that 
normally prevent the highly interconnected positive (excitatory) connections between 
neurons from running amok. Like an avalanche or a wildfire, one over-excited event can 
spread throughout the brain. H 

In 1953, at the age of 27, H.M. agreed to a highly experimental (and presum¬ 
ably desperate) procedure being done by William Scoville at Hartford Hospital in 
Connecticut: they would remove the part of the brain that contained the epileptic 
focus, the starting point of the epileptic avalanche. 42 In the subsequent procedure, 
Scoville removed both of H.M.’s hippocampi and much of the surrounding tissue 
(called the “medial temporal lobe”). 


G Traditionally, patients are referred to by their initials to preserve their privacy. When H.M. 
died in 2008, his full name was revealed. 36 But even before, known only as H.M., he was arguably the 
best-known brain lesion patient in neuroscience, rivaled only by Phineas Gage, whom we met in our 
discussion of frontal cortex and emotion (Chapter 8). 

H In highly interconnected tissue (such as the hippocampus and neocortex), excitatory recurrent 
connections (which encourage neurons to fire) are balanced by local negative recurrent connections 
(which discourage neurons from firing). If the excitation and inhibition are out of balance for a tran¬ 
sient moment, you can get runaway excitation leading to massive firing of neurons. 38 Interestingly, sei¬ 
zures can arise from too much transient excitation or too much transient inhibition. Brain function 
generally sits in dynamic balance, which makes it easier to respond quickly and makes the brain more 
flexible in its responses, but it also means that a transient event takes time to return to normal. The epi¬ 
leptic event is a phenomenon known in the physics and electrical engineering fields as “ringing”—the 
system gets pushed too far to one side, and as it returns to its base point, it bounces too far to the other 
side, much like a spring that oscillates when pulled or pushed. 

Thus epilepsy is incorrect, extraneous firing of neurons. Since the firing of neurons is memory, percep¬ 
tion, and cognition, this extra firing of neurons will be perceived by other parts of the brain as real. Epileptic 
seizures in the auditory cortex are often preceded by patients hearing music, events in temporal-lobe struc¬ 
tures can trigger memory, and seizures in emotional parts of the brain (such as the ventral frontal cortex) 
can trigger feelings. 39 Dostoevsky (a noted epileptic) described seizures in his novel The Idiot as being pre¬ 
ceded by an emotional moment of exceptional clarity of joy, hope, and vitality.® This emotional feeling of 
deep religious connection has been seen in some modern patients with temporal-lobe epilepsy as well. 41 
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This procedure was introduced by Wilder Penfield and Herbert Jasper at the Montreal 
Neurological Institute in the early 1940s. 43 It is still done today as a last resort for intrac¬ 
table epilepsy. 44 But today before any tissue is removed, doctors spend time locating the 
focal site (the starting point) of the epilepsy by recording for long periods of time (even 
up to weeks) to make sure that they localize the focus as tightly as possible, and then 
they spend time determining, as best they can, what that part of the brain does. In 1957, 
none of these techniques were available; all Scoville and his colleagues had available 
was the electroencephalogram (EEG), which doesn’t have enough resolution to break 
down the information being processed, and which was only good enough to show him 
that the epileptic focus was somewhere in the medial temporal region. So, desperate to 
stop the epileptic seizures, H.M. agreed to let the surgeon remove a large portion of his 
hippocampi and his medial temporal lobe bilaterally. 

The brain (and the mind) are physical things—removing apart of the brain removes 
a part of the computation. (Sometimes other parts can step in to accomplish a task in 
a different way, but sometimes not. This is the fundamental error in the software/hard¬ 
ware analogy.) At the time, in 1957, the computational role of hippocampus was still 
unknown. 

In terms of its immediate effect, the surgery improved H.M.s epilepsy dramatically 
and made it possible to control the epilepsy with medications. His general intelligence 
did not seem to be affected and his linguistic abilities remained normal, but the surgery 
left H.M. with a devastating memory impairment—he couldn’t remember anything 
new. While he could remember his past, new information could be remembered only 
as long as he attended to it. 45 If he was distracted, it was gone. This strange memory 
disability had been seen before and was known as anterograde amnesia.* 6 It was often 
seen in patients with Korsakoff’s syndrome, a disease caused by chronic alcoholism 
combined with a vitamin B (thiamine) deficiency. 47 Case studies of anterograde amne¬ 
sia are described beautifully in Oliver Sacks’ The Man who Mistook his Wife for a Hat 
(see “The Lost Mariner” and “A Matter of Identity”), in A. R. Luria’s The Man with a 
Shattered World, and (fictionalized into a complex mystery story) in Christopher Nolan’s 
Memento. 

H.M. was a patient who was normal (except for his epilepsy), of above-average intel¬ 
ligence, with a sudden loss of this very specific ability to learn new things. Here was a 
chance, in this terrible tragedy, to learn what went wrong and to understand how not 
to make this same mistake in the future. 48 Scoville called in Brenda Milner, a neuropsy¬ 
chologist who had studied with Donald Hebb and Wilder Penfield at McGill University. 
While Milner found that H.M.s inability to learn new memories was pervasive and 
crossed all subjects and domains (visual, auditory, linguistic, etc.), she also found that 
H.M. could learn a mirror-tracing task that took both H.M. and control subj ects several 
days to learn. 49 Along with her student Suzanne Corkin, they found that H.M. could, in 
fact, learn new tasks, but wouldn’t remember doing them. These tasks were all the kind 
that take lots of practice and are learned slowly, like mirror-writing or tracing intricate 
paths. Corkin and Milner described how H.M. would be incredulous that he could do a 
task because he could not remember that he had ever tried it before, yet the rate at which 
he learned these slowly learning, practice-requiring tasks was approximately normal. 

This led to the suggestion that there were two memory systems—one that learned 
stateable facts (“My car is parked at the airport.”) and another that learned skills (riding 
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a bicycle, throwing a baseball). Facts could be declared—I can tell you where my car is 
(“at the airport”) and you know it immediately. But skills are procedures that can only 
be learned through practice. No matter how much I tell you how to throw a baseball, you 
won’t be able to do it until you try it for a while. The first kind of memory was termed 
declarative, the second kind procedural. 50 

Subsequent work has suggested that declarative memory has two components, an 
episodic component including memories of specific places and times (the time I forgot 
my wife’s birthday) and a semantic component consisting of facts (the actual date of my 
wife’s birthday). 51 We now know that episodic memory is a constructed imagination of 
the past—it likely corresponds to the same system used to imagine the future. 52 This is 
the Deliberative system that we described earlier (Chapter 9). Semantic memory does 
not yet have an accepted correspondence in our decision-making taxonomy but may be 
similar to the situation-recognition and narrative components (Chapter 12). 

What is important here, and what dooms the hardware/software analogy, is that 
patients such as FI.M. are not shifting the software from one part of the computer to 
another. Rather, each component is able to perform certain calculations on the incom¬ 
ing information. When the lost mariner in Oliver Sacks’ case study learns to become 
familiar with his life among the nuns and finds peace in his garden, he has not found 
a way to regain his ability to learn new memories. Instead, he has found peace in the 
way he uses other systems to accomplish his tasks. When H.M. learned to accomplish 
mirror-writing, he learned it using the procedural memory system. 


Cortical structures—Can visual cortex take over 
for auditory cortex? 

H.M. lost his hippocampi and other specialized subcortical structures. 53 Subcortical and 
cortical processing are quite different: while subcortical structures tend to be special¬ 
ized, the cortex is a large two-dimensional sheet of repeated processing units, arranged 
in small, repeated columns. 54 

Current theories suggest that each repeated part of the cortex performs a similar 
computation on a unique set of inputs. (Exactly what that computation is remains 
controversial, but it seems to be some form of categorization process occurring via a 
content-addressable memory mechanism. 55 See Appendix C for a discussion of what is 
known about the computations occurring in the cortex.) However, that computation is 
fundamentally dependent on the inputs, and, unlike digital computers, those inputs are 
incomplete. 56 That is, you can’t send any input anywhere you want in the cortex. Each 
cortical column receives a limited set of inputs, and the information carried by those 
inputs is all that it has available. 

We can see the limitations of this by looking at how cortical representations can shift 
with changes in sensory inputs. This has been studied in the most detail in the context of 
primary sensory systems (particularly somatosensory [touch] systems). 57 Cells in these 
sensory areas receive inputs from a broad range of input cells, with stronger inputs usu¬ 
ally at the center and input strength falling off as the input changes. 58 We talk of “tuning 
curves,” which generally have a peak sensitivity but a broad range (see Appendix B). The 
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cells compete with each other to determine which signals will form the peak of each 
tuning curve. Take your palm, for example: at each point on your palm, a host of mecha- 
noreceptors have molecular mechanisms that detect touch. These detectors lead the cell 
to fire spikes, which are transmitted to the spinal cord and then through multiple steps 
to your sensory cortices, where they are interpreted as touch sensations. Even though 
these individual mechanoreceptors cover very large areas of your palm, you can localize 
a pinprick touch to a small point because your somatosensory cortex implements a sort 
of winner-take-all process using a mechanism called lateral inhibition, in which corti¬ 
cal cells inhibit their neighbors. This means that although lots of cells received input, 
only the ones representing the touched location remain active. This system integrates 
information from a very large area but provides a very accurate interpretation of the 
sensation. 

More areas of your somatosensory cortex are sensitive to your palm and fingertips 
than to your arm, which makes your palm and fingers a more sensitive sensor than your 
arm. A classic high school experiment is to close your eyes and have a lab partner touch 
two pencils to your hand or arm or back. Can you tell that they are two points, or do 
you feel it as one? (As a control, the experiment is usually done with the lab partner 
sometimes using one pencil and sometimes using two and then asking the subject 
“one or two?”) The sensitivity of your palm is much more accurate than your arm or 
back. This nonuniform distribution occurs in all sensory systems. Your auditory cortex 
spends more cortical area interpreting the frequencies at which human speech occurs 
than other frequencies, while your visual cortex spends more cortical area interpreting 
the center of your vision (called the fovea) than the periphery. 59 

But what happens when one loses part of the input? Because the cells are actually 
receiving input from large areas of the sensory field, but the far inputs are usually com¬ 
peted out, if the central part of the input is lost, the far inputs will now win the com¬ 
petition, and the cell will shift the area it listens to and show responses to the areas 
that remain. 60 Notice that although the compensation is a change in what information 
is processed by the cortex, the compensation is not due to a shift in software to the 
no-longer-used cortex; rather, it is due to a shift in which already-wired-up inputs are 
being listened to. 

This effect produces interesting consequences, particularly in the sensations in phan¬ 
tom limbs. Patients who have lost a limb often feel sensations in that vanished limb. 
V. S. Ramachandran has shown that these sensations are due to sensory stimuli arriving 
at structures represented nearby on the somatosensory cortex. 61 The layout of the pri¬ 
mate somatosensory cortex has been known since the late 1800s. 62 It has some strange 
discontinuities—for example, the fingertips are represented near the face. Patients miss¬ 
ing a hand will sometimes feel sensation on their hand when their faces are touched. 
Patients with amputations often complain of pain or itching in their phantom limb; 
these sensations are extremely uncomfortable because they cannot scratch a limb that 
isn’t there. But scratching the adjacently represented area can sometimes relieve the 
phantom limb pain. 

If the inhibition between areas is incomplete or disinhibited or the connections are 
incompletely pruned or overconnected during development, these connections can 
cross between sensory systems and lead to experiences of relationships between sen¬ 
sations—one can hear colors or see sounds. 63 This process, called synesthesia (from syn, 
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meaning “with” or “together,” and asthesia, meaning “feel” or “perceive” [as in anesthesia 
or aesthetics]), is a fascinating interaction between sensory systems. Intriguingly, these 
relationships are not random; they tend to be constant within a given person. Flavors 
taste like shapes. Numbers have associated colors. 64 

There is some evidence that during development, cortical representations can shift 
more than during adulthood. 65 In part, this seems to be because cortical connectivity in 
juveniles and children is broader and gets pared down as cortical systems develop. This 
has been studied most quantitatively by Eric Knudsen, looking at how representations 
in the colliculi of owls change in response to sensory changes. The superior and inferior 
colliculi are subcortical brain structures in birds and mammals that perform an atten¬ 
tion or targeting function—the cells are organized in a topographic manner represent¬ 
ing position around the head of the animal, horizontally and vertically. 66 Stimulation 
leads to the animal attending to that horizontal and vertical position. The cells in the 
inferior colliculus respond primarily to auditory (but also more weakly to visual) input, 
using auditory cues (volume and time difference between the two ears) to localize a 
target in space, while cells in the superior colliculus respond primarily to visual (but also 
more weakly to auditory and tactile) input. When you hear a footfall behind you and 
whirl around to see what it is, that’s your colliculi at work. The inferior colliculus derives 
from the auditory tectum of reptiles, while the superior colliculus derives from the optic 
tectum in reptiles, which are their primary sensory systems. In birds and mammals, both 
the inferior and superior colliculi sit underneath evolutionarily newer structures, the 
auditory and visual cortices. 67 Because these colliculi are organized topographically in 
terms of an output, the multiple sensory systems have to be co-aligned. Likely because it 
is easier to calculate an orientation from visual cues than from auditory ones, the align¬ 
ment of both (even the inferior colliculus) turns out to be based on visual signals. 68 

Eric Knudsen and his students used this phenomenon to study how the inferior col¬ 
liculus of barn owls changes when the visual inputs are changed. 69 Owls are particularly 
good at localizing sounds in three-dimensional space (since they usually hunt at night). 
An owl can track the footfalls of a mouse from meters away, target in on it, and hit it 
accurately. 70 As the owl is developing, it needs to learn to align the spatial information 
it derives from auditory cues with the spatial information it derives from visual cues. 
Knudsen and colleagues fitted owls with prism glasses that shifted the visual signals to 
the left or the right by a certain angle. 1 In the owl, the auditory system shifts to align 
to the visual, so the auditory inputs to the inferior colliculus had to arrive at slightly 
different locations. 

What Knudsen and colleagues found was that, initially, the auditory inputs to the 
inferior colliculus were very broad, but that they were pruned away as animals progressed 


1 These prism glasses may be familiar to some people visiting science museums. A typical neurosci¬ 
ence game shown at science museums and during presentations to school students is to have a volun¬ 
teer toss a beanbag into a basket with prism glasses on. Because the prism shifts the visual world to the 
left by some angle (say 20 degrees), the person typically misses the basket by 20 degrees in the opposite 
direction. With continued tries, the person can learn to compensate over the course of a few minutes. 
And then, when the prisms are removed, there is an after-effect where the person misses the basket by 
20 degrees in the shifted direction, which takes a few minutes to correct. With more experience with 
and without prisms, the time it takes to shift decreases. 71 
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through development. In particular, the owls passed through a specific age, called a 
sensitive period, after which the auditory inputs crystallized into place and became no 
longer malleable. 72 Manipulations of the prisms before the crystallization were much 
more effective than manipulations afterwards. 

This sort of sensitive period is seen in many systems, including the visual system, 
the auditory system, and others. Similar sensitive periods are seen in cats and monkeys 
learning to see, birds learning to sing, and human children learning language. 73 This is 
why children are much better at learning foreign languages than adults. The timing of 
the sensitive periods is due to a physiological change in the brain, ending when plasticity 
crystallizes in different brain structures at different ages. 74 

But, again, just as was seen in the phantom limb results, above, the shifts that the owls 
could accommodate behaviorally completely matched the extent of the physical chang¬ 
es. 75 The shifts available depended on the connectivity changes. Similar dependences 
have been seen in somatosensory systems and in auditory systems. 76 Sensory changes 
reflect actual changes in physical connectivity within the brain. But what about memo¬ 
ries? Can’t information be translated from short-term to long-term memory? 


Transferring memories from one part of the brain to another 

On recovery from his surgery, H.M. showed a new and profound deficit in his ability 
to learn new memories. However, H.M.’s earlier memories seemed to be intact. Older 
memories were better represented than more recent memories. 77 At the time, this was 
taken to imply that memories were transferred from a short-term memory store to a 
long-term memory store (like information being written from your computer memory 
to a hard disk or to a CD). But further studies have shown that these older memories are 
different from the more recent memories. 78 Something is changed in the transfer. 

Long-term memories tend to be very semantic, stored as facts, as narratives, as scripts 
and stories, while short-term memories tend to be episodic, with a personal “I was there” 
emotion to them. 79 This does not mean that we don’t have long-term memories with 
strong “I was there” emotions, but in H.M. and other similar amnesic patients, those 
older episodic memories vanished along with the short-term memories. 80 What seems 
to happen to memories is that, in the short term, they are stored as episodic events, but 
with time, they become relegated to a semantic storage that is fundamentally different. 
All of H.M.’s long-term memories, for example, were semantic descriptions and did not 
contain the episodic descriptions that we normally associate with important long-term 
memories. 

Howmightthis difference between semantic and episodic memory arise? Throughout 
this book, I’ve tried to explain mechanism with theories that can actually explain how 
the information level arises from the physical. In this case, we require the answers to 
four questions: (l) How are episodic memories stored? (2) How are semantic memories 
stored? (3) How is information transferred from one to the other? In addition, it would be 
nice to understand (4) Why is such a mechanism evolutionarily useful? 

We have encountered episodic memories elsewhere in this book (in Chapter 9). We 
noted that they were not stored flawlessly, but rebuilt from components stored in dif¬ 
ferent cortical areas. 81 The structure tying those components together seems to be the 
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hippocampus, which receives input (through the entorhinal cortex) from pretty much 
the entire neocortex, and which sends output (again through the entorhinal cortex) 
to pretty much the entire neocortex. 82 We have also encountered semantic memories 
elsewhere in this book (in Chapter 12). 

Why would we need these two different systems? Presumably, as with the other 
examples of multiple decision-making systems that we’ve encountered so far, each 
mechanism has advantages and disadvantages. By using the right one at the right 
time, we can access the advantages of each and reduce the disadvantages. One theory, 
proposed by Jay McClelland, Randy O’Reilly, and Bruce McNaughton in the 1990s, 
was that we need two learning systems—one system that could store a few memories 
quickly by making sure the representations were really separable, and another system 
that could store lots of memories, but took the time to store them slowly so that they 
wouldn’t interfere with each other. 83 The theory arose from work in the computational 
aspects of content-addressable memories (see Appendix C), in which it was observed 
that storing memories sequentially in a neural system where memories were distributed 
across many neurons produced catastrophic interference —as one memory was stored, 
older memories were lost. 84 

Because these systems stored information in a distributed manner, through 
small changes in individual connections, each connection between neurons (each 
synapse) participated in many memories. This meant that if you stored a memory, 
the changes in synapses needed to store the new memory could undo some of the 
changes needed for the previous memories. This is a process called interference — 
new memories interfere with the old ones. 85 On the other hand, if you interleaved the 
storage of the memories, changing the synapses a little bit toward what they would 
need to be to store the first memory, and then a little bit toward what they would 
need to be to store the second, the synapses could find a way to store both memories 
simultaneously. 

Effectively, there is a tradeoff between being malleable enough to learn new things 
quickly and stable enough to hold memories for a long time. McClelland, O’Reilly, and 
McNaughton proposed that if you had a system that could store memories quickly, 
without interference, then you could use that fast-storing system to interleave the mem¬ 
ories into the slower, longer-term storage. They proposed that the hippocampus, which 
contains mechanisms to reduce interference between stored memories (and thus can 
store memories quickly), served as the fast storage and the cortex (which learned more 
slowly and contained many more synapses) served as the slow storage. 86 Although this 
theory does seem to be primarily correct, as we’ve discussed in this chapter, the transfer 
is a transfer, not just of information but also of kind, from an episodic representation to 
a semantic representation. 87 

This means that the brain needs to transfer a memory from a quickly learned sys¬ 
tem which reconstructs it from parts into a slowly learned storage system in which it is 
stored in the strength of connections between neural structures. 88 Before we look at how 
this transfer occurs, it is important to note that the evidence is very strong that these 
are two separate systems, each of which can learn memories separately. Although learn¬ 
ing generally proceeds from quickly learned episodic memories to more slowly learned 
semantic memories, if structures critical for the episodic memory systems are damaged, 
the semantic memory system can still learn. 89 For example, patients with hippocampal 
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damage are impaired at episodic future thinking but are still able to learn to recognize 
familiar scenes that they experience multiple times and are still able to construct narra¬ 
tives. 90 Just such an example can be found in Oliver Sacks’ lost mariner, who found a way 
to become comfortable with his new garden but remained unable to ever learn explicit 
(episodic) memories. 91 Suzanne Corkin says that through her long interaction with him, 
H.M. began to recognize her and thought he knew her from high school. 92 This con¬ 
struction of an explanation is sometimes called confabulation 93 and is a consequence 
of the construction of narratives (Chapter 12) through content-addressable-memory 
processes (Appendix C). 

Although it is possible for semantic memories to be constructed through extended 
experience, it is also possible to take a single experience (encoded in episodic memory) 
and, through internal repetition, transfer it to a semantic memory. This transfer seems 
to occur primarily during sleep. 94 Scientists have known for a long time that sleep is a 
critical part of the learning process. Not only do animals and humans deprived of sleep 
not learn well, but they also do not easily remember the tasks that were learned before 
they were deprived of sleep. 95 

One of the most interesting phenomena discovered over the past couple of decades 
is a phenomenon called “replay,” in which neural firing patterns seen during behavior 
replay themselves during sleep afterwards. 96 The phenomenon was originally seen in 
neural recordings of the hippocampus but is now known to occur throughout the 
hippocampus, neocortex, and some subcortical structures. Because it is so hard to 
record neural signals from humans, the replay phenomenon has generally been stud¬ 
ied in rats and monkeys, but the timing of the replay events corresponds nicely to 
when sleep-deprivation studies have found that sleep is critical to consolidating 
memories. 97 

Reactivation of behavioral neural patterns during sleep was first seen in 1989 by 
Constantine Pavlides and Jonathan Winson, who recorded from pairs of hippocam¬ 
pal cells from rats. 98 Hippocampal cells in rats (place cells) have the very convenient 
tuning function that they respond only when the rat is in a specific position in an 
environment (the place field of the cell). Each cell has a different place field. 99 Pavlides 
and Winson chose place cells with nonoverlapping place fields; thus, they were able 
to expose the rat to the place field of one cell while not exposing it to the place field 
of the other. They then found that during sleep after the behavior, the cell with the 
place field the animal had been exposed to fired much more than the cell with the 
other place field—even though the animal was not in either place field when allowed 
to sleep. 

Our ability to observe reactivation and replay took a giant step forward with the 
development in 1993 of multi-tetrode recording technology by Matthew Wilson and 
Bruce McNaughton, who brought together several technologies and were able to 
record from almost 150 cells simultaneously from their behaving rats. 100 Because each 
place cell has a different place field, from the activity across the set of cells, Wilson 
and McNaughton were able to decode the position of the rat during behavior from 
its neural ensemble. (See Appendix B for a description of how this decoding pro¬ 
cess works.) From a large neural ensemble, one can also determine whether patterns 
observed during behavior repeat (reactivate) afterwards during sleep. 101 From a large 
neural ensemble, one can even determine whether sequences observed during behavior 
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repeat (replay) afterwards during sleep. A host of studies over the subsequent years, 
starting from Wilson and McNaughton’s follow-up paper in 1994 and culminating in 
dozens of papers in the past decade, have shown that what is reactivated is actually the 
full neural pattern—cells that were coactive during behavior remain coactive during 
reactivation, while cells that were not coactive are not, and that what is replayed is the 
actual sequences observed by the rat: cells reactivate in the same order during sleep as 
during behavior. 102 Control studies have shown that this reactivation is a consequence 
of the behavior—it does not occur during sleep before the behavior, only afterwards. 
Reactivation and replay are also seen in neocortical systems during sleep as well. As 
with the hippocampal reactivation and replay, the same neural patterns observed dur¬ 
ing behavior appear afterwards during sleep. Reactivation and replay have been seen 
in a host of cortical systems, including the prefrontal, parietal, and even primary visual 
cortex. 103, 'J 

At this point the computational and neurophysiological function of replay is 
unknown. 106 Is it to aid storage of information within the structure itself (hippocampus 
or cortex)? 107 Is it to transfer information from the hippocampally based episodic sys¬ 
tem to the cortically based semantic system? 108 Is it to erase old memories from the hip¬ 
pocampus while enhancing new ones? 109 While these questions are still being addressed 
by researchers today, we have intriguing hints that all of these functions may be impor¬ 
tant. What is known is that blocking replay, either pharmacologically (through chemi¬ 
cal means) or electrically (by stimulating the hippocampus whenever a replay event is 
about to happen K ), disrupts memory retention. 111 It is also known that replay events in 
hippocampus tend to correspond to reactivation events in cortical structures, and that 
after replay events, neural patterns in cortical structures become more tightly coupled, 
even between cortical structures, exactly as would be predicted by the transfer and inte¬ 
gration hypothesis. 112 

Throughout this discussion, I have avoided calling these replay events “dreams,” 
because we don’t know if these replay events seen in animals correspond to what we 
experience as dreams. (We can’t ask the animal if it is dreaming, and it is very dif¬ 
ficult to record from humans during dreams.) But, of course, it is very likely that this 
reactivation/replay phenomenon being studied in animals is the physical instantia¬ 
tion of the phenomenon we refer to as dreams. Dreams are often jumbled sequences 
of past experiences. 113 Although the animal experiments report actual replay of direct 


t Replay in nonhippocampal subcortical structures, such as the dorsal system in the basal ganglia 
(Chapter 10), has not yet been reported, but there has not been the same level of extensive study 
done on these structures during sleep. The one exception is that the ventral striatum (the nucleus 
accumbens, which we saw was involved in representing potential outcomes during deliberation; see 
Chapter 9), replays reward-related activity in conjunction with the hippocampal replay of experienced 
sequences. 104 Sleep is also critical for procedural learning, but whether this is due to replay events in 
dorsal basal ganglia systems remains, as yet, unknown. 105 

K Replay events in the hippocampus tend to occur during an identifiable local field potential (LFP) 
event called a ripple or a sharp wave (sometimes called a “sharp-wave-ripple-complex”) that is observ¬ 
able in the hippocampus. Because these LFP events can be detected quickly, they can be disrupted as 
soon as they start through a short electrical stimulation of the hippocampus. 110 
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events, in part this may be because those sequences are the easiest to recognize. Some 
experiments have found interleaved representations of old and new experiences in 
rats during REM sleep. 114 During awake, resting behavior, recent experiments have 
found “replays” of sequences that the animal has never actually experienced, such 
as the sequence experienced by the animal but backwards, or chains of experi¬ 
enced sequences that share commonalities but have not actually been experienced 
together. 115 

In a very intriguing experiment, Robert Stickgold and colleagues trained a popula¬ 
tion of patients with hippocampal amnesia (like H.M.) to play the videogame Tetris. 116 
These patients can learn tasks using their nonhippocampal memory but do not remem¬ 
ber that they have played the game before. When Stickgold and his colleagues woke 
these people up from sleep and asked them about their dreams, they reported seeing 
strange shapes falling from the sky but were terrified because they had no idea where 
these shapes were coming from. It seems likely that the cortex was replaying the sen¬ 
sory stimuli it had seen (falling shapes) but that, without a hippocampus, these patients 
could not integrate these events into their episodic memories or remember where these 
images came from. 


Summary 

So even after lesions, animals (including humans) can learn new tasks, but these new 
tasks are learned in different ways, using different decision-making systems. Similarly, 
intact cortical structures can take over for damaged cortical structures, but only to the 
extent that input connections are available. And memories can be transferred from 
hippocampal-dependent episodic structures to cortical-dependent semantic processes, 
but they are modified in that transition. The brain is a physical machine. The mind is 
not software that happens to be implemented on the brain’s hardware, but is directly 
dependent on the processes of the physical brain. 

The mind used to be thought of as a hiker through a forest, or a surfer on a wave, or 
perhaps as the wave itself flowing through a physical ocean, or as something affected 
by but separate from the brain itself. 117 Historically, this separation was based on 
Descartes’ dualism between the physical and cognitive entities. 118 In the modern cogni¬ 
tive science of the past century, this separation was based on the new theories of digital 
information. 119 

However, as we have seen in this chapter, the theory that mind and brain are sepa¬ 
rable is untenable, and the available data suggest instead that they are the same thing. 
The evidence for mentation as the processing of information is overwhelming, but the 
different components of the brain process that information differently. It is not a wave 
flowing through an open ocean, capable of traveling in any direction, but rather a wave 
traveling through a series of highly restrictive canyons. This has implications for every 
aspect of our mental lives, from mental processes like imagination to diseases and men¬ 
tal disorders like Parkinson’s disease, Alzheimer’s disease, post-traumatic stress disor¬ 
der, and addiction, to mental constructs of behavior like craving, impulsivity, free will, 
and morality. 
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Imagination 


I want to walk on the beach with you, 
feel the cool ocean pull at our feet, 
memory like water 
filling our footprints in the wet sand. 


If the brain is the physical instantiation of the mind, then every 
mental event must be reflected in the physical realm, from imagi¬ 
nation to hallucination to mental imagery. The current theory in 
neuroscience is that this physical realm is the firing of neurons — 
that is, just as we know that the firing of sensory neurons responds 
to sensory cues, and the firing of motor neurons controls muscles, 
this theory predicts that nonsensory mental (cognitive) events are 
not just reflected in but actually are the firing of neurons. 


The fact that the cortex can be better understood as a two-dimensional sheet of repeat¬ 
ing columns rather than as a three-dimensional structure suggests that we can think of 
the cortex as a series of interconnected maps. Visual structures map the visual field in 
front of you, auditory structures map tones and sounds, and both sensory and motor 
structures map your body . 1 

In the mammal (including the human), the primary visual cortex happens to be at 
the back of the brain. The axons connecting your eyes to your visual cortex have to run 
all the way across the brain (they run underneath), which is why a concussion (in which 
your brain slams against your skull like an unconstrained obj ect in a suddenly stopping 
car) often leads to visual problems . 2 Recordings of neurons from primary visual cortex 
find that for the most part, the cells in each visual cortical column are tuned to the 
same point on the retina. Some of these cells are tuned to spots on the retina, while 
others are tuned to oriented lines, and others are tuned to moving oriented lines. Some 
are excited by light at the location, while others are inhibited. But each column in the 
primary visual cortex is tuned to light arriving at one spot on the retina . 3 

Connectivity in the cortex is based on a center-surround architecture, in which cells 
representing similar locations on the retina are more tightly coupled than cells repre¬ 
senting distal locations, with a larger-scale inhibitory connectivity. There are also some 
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longer-spaced connections that have subtle effects (they will be particularly important 
when we discuss migraines and the fortification hallucination), but the primary connec¬ 
tivity seems to be between adjacent columns representing similar information . 4 At the 
simplest level, cortex can be thought of as a sheet of local excitation and broader inhibi¬ 
tion. This connectivity produces what is known mathematically as a local coupling kernel, 
a connectivity repeated at every location . 5 Such coupling kernels are seen in physical 
objects as well, such as the relationship between the coils of a spring or the physical 
connectivity of a guitar string (in one dimension), the physical connectivity of a bed 
sheet (in two dimensions), or the relationship between molecules of water (in three 
dimensions). 

Those are three particularly interesting examples, because they all show wave effects. 
We have all seen waves on water, and we have all snapped a bed sheet flat, sending a 
wave across it; a classic junior high school physics experiment is to take a spring and 
send a traveling wave through it. And, of course, the sound generated by a guitar string is 
based on a standing wave, as it vibrates under the restricted conditions (mathematically 
termed the boundary conditions ) that the two ends are held fixed and cannot move. Just 
as waves can travel across physical objects, so too can waves of neural activity travel 
across the cortical sheet . 6 

What would it feel like to have a wave of activity travel across your visual cortex? 
If neural activity in visual cortex is visual perception, then you would perceive a wave 
across your visual cortex as a wave of light traveling across your vision. In other words, 
you would hallucinate a wave of light. But people have been reporting their hallucina¬ 
tions for thousands of years, and random hallucinations don’t appear as a wave of light 
across your vision . 7 

The reason for this is that the primary visual cortex in the primate (including the 
human) does not represent the retinal input with a one-to-one transformation. Instead, 
there is more representation of the center of your retina (an area called the fovea) and 
less representation of the periphery . 8 The specific mathematical description for this 
transformation is called a log-polar transformation. Think of your visual cortex as two 
rectangles pasted onto the back of your brain, lying next to each other. Each side of 
your visual cortex represents half your visual field. Cells in the fovea project (through an 
intermediate thalamic structure that does additional processing A called the lateral genic¬ 
ulate nucleus) to cells in the visual cortex near the midline, while cells in the periphery 
project (again through intermediate structures) to cells in the visual cortex more later¬ 
ally. Moving along the horizontal axis of the visual cortex is equivalent to moving out 
from the fovea. Just as there are more cells in the fovea than the periphery, there is more 
visual cortex dedicated to the fovea than the periphery. Similarly, moving along the ver¬ 
tical axis is equivalent to moving in a circle around the fovea, with the top of your visual 
cortex corresponding to the top of your visual field. 

This means that a wave that was perfectly aligned to the vertical axis of your visual 
cortex, one that started at the midline and progressed away from the midline, would 
appear as an expanding ring. Because more and more visual field is represented by the 


A A lot of processing is done in the retina as well. This book, however, is not about how the brain 
processes sensory information, and so I won’t go into those details, but a good resource for anyone who 
is interested is the recent textbook by Chaudhuri. 
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same-size area of cortex as one moves out to the periphery, a traveling wave moving at 
a constant speed on the cortical surface would appear to speed up as it travels to the 
periphery. 

Simple visual hallucinations are often seen when falling asleep and waking up, under 
certain flickering lights, and in sealed dark rooms. They are also seen under the influence 
of certain recreational chemicals (drugs). These simple visual hallucinations are very 
similar to those painted on and carved into sacred stones and cave walls. People have 
been reporting their chemically-induced hallucinations for thousands of years. They are 
also the same patterns often seen in “near-death” experiences. 9 

These images tend to be made of radial lines, spirals, and spider-web shapes. They 
often appear as spiral checkerboards, honeycombs, or dots. And sometimes they are as 
simple as a white light in the center of the vision. Sometimes the images are stable and 
sometimes they move in a spiral inward or outward pattern. The images move with the 
retina (and are thus seen in “retinal space,” which is particularly strange to experience 
because we are used to seeing things in “allocentric” or real-world space, which moves 
on our retina when we move our eyes). These images are the visual interpretation of 
waves in the cortex. 

A vertical wave flowing from top to bottom along the visual cortex will appear as 
rotating radial lines. A collapsing wave flowing from the lateral to medial aspects will 
appear as a collapsing light, and may easily be interpreted as a tunnel. A wave that is 
diagonally oriented along the visual cortex will appear as a spiral. As the wave moves, 
the spiral can appear to spiral inward or outward. See Figure 17.1. 

The primary visual cortex in the primate is actually connected up in a more com¬ 
plex manner than a simple center-surround kernel. 10 The cortical columns are arranged 
into local hypercolumns, which represent a single location on the visual field, but 
contain orientation-selective cells, which respond to lines oriented in different ways. 
A hypercolumn contains columns of cells representing each potential orientation. Each 
column is coupled to cells within its own hypercolumn (because they represent the 
same location in the visual field) and to nearby columns in other hypercolumns that 
share its preferred orientation (because they represent lines of the same orientation). 
The cross-hypercolumn connections seem to be laid out in an approximately hexagonal 
grid. It is the hexagonal grid across hypercolumns that produces the checkerboard or 
honeycomb components of some visual hallucinations. 

The visual system in the primate (including the human) is very complex, including 
a host of separate cortical areas, each of which is dedicated to a different useful feature, 
such as colors, spatial locations in the visual world, even faces, rooms, and other dedi¬ 
cated objects. 11 Current theories suggest that this complex network of visual fields is a 
set of separate, dedicated feature detectors, tuned to the set of natural images we experi¬ 
ence in our lives. 

Hallucinations of more complex events and objects are likely to depend on the fir¬ 
ing of other cortices, such as those representing objects, faces, locations, and other 
sensations. However, because we don’t know how these higher-order representations 
are encoded, we don’t know how waves would appear on them. Random firing in the 
visual cortex is also seen under hallucination conditions, which appear as small colored 
spots on the visual field (again, that follow your retina as it moves). 12 Just as the firing 
of the primary visual cortex is perceived as vision, the random firing of neurons in the 
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Figure 17.1 The log-polar transformation in the primate visual cortex. Patterns on the 
visual cortex (horizontal lines, vertical lines, diagonal lines) are transformed into retinal 
perceptions (radial lines, concentric circles, a spiral) because of the log-polar transformation 
from the retina to the visual cortex. Similar patterns are seen during sensory deprivation, 
drug-induced hallucinations, and near-death experiences. Similar patterns also appear drawn 
on ancient petroglyphs. 


face-representation area (a small, dedicated piece of the cortex called the superior tem¬ 
poral sulcus or STS) 13 would be perceived as seeing a specific person. 


Migraine and the fortification illusion 

One of the best-described visual hallucinations occurs during migraines and is described 
as the “fortification” hallucination or “visual aura.” 14 Generally, these are experienced 
as extremely bright jagged edges, progressing slowly across the visual field, followed 
behind by a visual scotoma or blackness, obscuring one’s vision. Interestingly, the line of 
jagged edges speeds up as it progresses from the center of one’s vision to the edge, over 
the course of 10 to 20 minutes. At this point, you may be able to guess why the jagged 
edge speeds up as it progresses out from the center to the periphery of vision. 

These “fortification” auras are directly explainable as the firing of cells in the visual 
cortex. 15 As noted above, cells in the visual cortex represent visual information by being 
tuned to short, oriented lines. Different orientations within a hypercolumn are con¬ 
nected to each other. Thus, if much of a hypercolumn was active, it would appear as a set 
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of angular lines, at sharp orientations to each other. Because the columns are connected 
across hypercolumns, one sees a continuous hexagonal wall. As the wave progresses 
from the medial edge (the center of one’s vision) to the lateral edge (the periphery of 
one’s vision), the wall seems to move. It is exceptionally bright because all of the cells 
are firing as the wave progresses through them. Firing uses up cellular resources, and 
cells become unable to fire (become refractory ) afterwards. 16 The inability of cells to fire 
behind the wave appears as a blank darkness or a scotoma. The wave appears to speed up 
because our cortex represents the center of our visual field with more cortical area than 
the periphery. Thus a constant-speed wave in the visual cortex covers more and more of 
our visual field as it progresses laterally and appears to speed up as it moves. 

The term “fortification” is an allusion to St. Hildegard von Bingen, a 12th-century 
nun, living in Germany, who left descriptions of visions of lights more brilliant than the 
sun progressing through her vision, which turn into black coals, of visions of a spreading 
fire of God, followed by gloomy darkness, leaving ethereal stars behind, and of visions 
of a head of fire scourging the darkness. 17 Several authors in the 20th century have sug¬ 
gested that St. Hildegard’s visions were migraine auras. 18 The jagged edges of the forti¬ 
fication illusion do, in fact, look like crenellations and battlements that might be seen 
atop a medieval castle. They are often brilliantly bright (being the firing of many cells 
simultaneously) and followed by a darkness (as the cells shut off and become refractory, 
tiring out and becoming unable to fire), which obscures vision (because vision is the fir¬ 
ing of specific cells; which subset of cells fires is the information about what you see—if 
all the cells are silent, one sees only darkness). Figure 17.2 shows a modern depiction of 
the fortification illusion. Flildegard explicitly described that the visions occurred during 
normal wakefulness, not during sleep or trances, and that she could not see the length 
or breadth of these visions (presumably because the images were on her visual cortex, 
so they moved with her vision when she moved her eyes). One can easily imagine the 



Figure 1 7.2 Fortification illusions in migraines, (a) Migraine sufferers see scintillating 
zigzags followed by darkness. A migraine sufferer watched a flickering display and drew 
what was seen, (b) The subject was asked to focus at a single point and draw the edges of 
these fortifications on a piece of paper at 2-minute intervals. The results reveal that the 
fortifications expand outward at accelerating speeds. From Grusser (1995), reprinted with 
permission from the publisher. 
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effect on a young woman in the 12th century of an inexplicable jagged bright line lead¬ 
ing an area of darkness expanding through her vision, especially one that continued 
whether her eyes were open or closed, that she could not look away from, and that was 
accompanied by a blazing, inescapable headache. 


Imagination 

If waves in the cortex and the random firing of cells produce hallucinations and the illu¬ 
sion of seeing things that are not there, what about the other direction? When we imag¬ 
ine something, is that also a physical process? This has been most thoroughly studied 
through visual imagery, in which people are asked to imagine a visual image. 19 

The first study to directly address this question was an examination of the abil¬ 
ity to mentally rotate images in ones mind. In a now-classic experiment, Roger 
Shepherd and Jacqueline Metzler asked subjects to determine whether two pictures of 
three-dimensional objects were pictures of the same object rotated through space, or 
whether they were different objects. 20 What they found was that the time it took a per¬ 
son to determine if the two objects were related was a linear function of the rotational 
angle between the two objects. They concluded from this that people were mentally 
rotating the objects (at a rate of about 60 degrees per second). Of course, when we look 
at the two objects and try to compare them, we have the subjective feeling that we are 
mentally rotating them (at least I do!). These rotations appear to activate visual and 
motor structures, as if people were imagining rotating the objects manually. 

Following up on this work, Stephen Kosslyn and his colleagues began testing the psy¬ 
chophysics® of mental imagery. 21 If I ask you to imagine a pirate boat and then to talk 
about the prow of the boat (What is the figurehead on the prow of the boat?), you are 
faster to answer questions about the prow than about the stern (Flow many guns are 
pointing aft?). Somehow distances matter in the mental imagery. These visual representa¬ 
tions have been validated by fMRI results that have shown that mental imagery activates 
the appropriate sensory areas of the brain: visual mental imagery entails activity in the 
visual cortices and auditory mental imagery entails activity in auditory cortices, etc. 

Of course, there is more to the brain than the primary sensory cortices. Normal 
people know the difference between imagining a boat and seeing one. This means that 
somewhere in the brain (likely the frontal cortices), other neurons are firing that differ¬ 
entiate them. One theory about hallucinations is that hallucinations entail mental imag¬ 
ery without those other neurons saying, “this is just your imagination—it’s not real.” 22 

A particularly interesting example of this sort of mental imagery has captured the 
imagination of neuroscience and psychology recently—motor control neurons in the 
monkey premotor cortices (the motor-related planning cortices 23 ) show similar activity 
whether the animal is performing the movement itself or whether it is watching another 
primate perform the movement. 24 These neurons have been called “mirror” neurons; 
however, it is not clear at this point whether they are a special type of representation in 
the primate cortex or whether they are simply reflecting the animals imagination of the 
others actions in its own motor-control neurons. 25 We have encountered these mirror 


Psychophysics is the study of the timing of perceptual responses. 
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neurons previously in Chapter 14, where we discussed them in terms of our ability to 
empathize and sympathize with others. 


Measuring mental imagery 

Mental imagery is the firing of neurons in the brain. This means that we might be able 
to identify a covert cognitive process if we recorded enough neurons and identified 
manipulations of a covert variable. 26 

The first study to do this was done in the 1980s by Apostolos Georgopoulos and his 
colleagues, who trained a monkey to reach toward a target. 27 Initially, the monkeys were 
trained to touch a center light and hold it until the light went off, at which time one of 
eight surrounding lights would turn on. The monkey was simply to reach to one of the 
surrounding lights for a juice reward. What Georgopoulos and his colleagues found was 
that neurons in the primary motor cortex were tuned to the direction the animal was 
reaching and that from a large set of neurons (thousands 0 ), they could decode the direc¬ 
tion the animal was reaching (or was about to reach). 29 They then trained the monkey 
to reach not to the lit-up light on the outer circle, but rather 90 degrees counterclock¬ 
wise to it. When they decoded the neurons, they found that the ensemble originally 
decoded to represent the direction of the lit-up light and rotated (through the inter¬ 
mediate directions) to the direction the animal was going to reach before the animal 
actually began its movement. 30 The animal was mentally rotating the reaching direction 
from the target to the cue. D In fact, the speed of that mental rotation in the monkey 
motor cortex matched the speed of mental rotation found by the classic Shepherd and 
Metzler study in humans from two decades earlier in which humans matched images of 
three-dimensional blocks. 

If it is possible to decode covert cognitive and imagined information from neural 
ensembles, then it should be possible to record the neural activity from the brain, to ask 
subjects to imagine moving their arms, to decode the neural activity of how they want 
their arms to move, and to move robotic arms to match. This is the promise of brain- 
machine interfaces and sounds like science fiction, but it is already a reality in limited 
conditions. 32 

Using the same motor-tuned areas that Apostolos Georgopoulos recorded from two 
decades earlier, Andy Schwartz (then at Arizona State University, now at the University 
of Pittsburgh) trained a monkey to track a ball in a virtual-reality three-dimensional 
space. 33 Wearing specially designed goggles, the monkey saw two balls—a yellow ball 


c At the time, only one neuron could be recorded at a time. Georgopoulos and his colleagues 
trained the monkey so well that each reach from center out was almost exactly the same motion. 
Recording one or two neurons per day, they then combined all of the reaches the monkey made over 
the course of several years to create a “pseudo-ensemble” of thousands of neurons. Their experiments 
have more recently been replicated using smaller sets (dozens) of simultaneously recorded neurons, 
but the conclusions have not changed. 28 

D This mental rotation by the monkey between the immediate reaction of reaching to the cue 
and the later correction to reach 90 degrees off from it is a special case of the issue we discussed in 
Chapter 10 that the brain first plans the quick and obvious reaction but can correct it if given time. 31 
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that appeared at a random point in three-dimensional space, and a blue ball that it 
controlled. If it could get the blue ball to catch the yellow one, it would get a juice 
reward. The secret (for which the monkey was never explicitly trained) was that the 
movement of the blue ball was not based on the movement of the monkey’s hand but 
instead on the decoded information from the monkey’s brain. When he presented his 
data, Andy Schwartz used to show a video in which the monkey initially tracked the 
virtual ball with its hands, but then as it realized that the tracking was moving faster than 
its hands, it dropped its hands to its sides and continued tracking the ball using only its 
mind. For obvious reasons, this video was affectionately referred to in the neuroscience 
community as the “Jedi Monkey video.” 

More recently, it has become possible to decode this imagined information to move 
robotic arms, not just in monkeys but now also in humans. 34 The work on this has been 
slow because it required two separate breakthroughs to work—from neuroscience, it 
required decoding algorithms that could identify what the arm was supposed to do, and 
from robotics, it required reliable robot arms that could be controlled with sufficient 
accuracy and simplicity. But both of those breakthroughs have been accomplished and 
several labs have now driven robotic arms from mental imagery. In 2008, a monkey in 
Andy Schwartz’s lab was able to feed itself using a robotic arm. 35 John Donoghue’s lab 
has shown that tetraplegic patients (who are paralyzed from the neck down) can guide 
robotic hands to move and to open and close. 36 In one of the videos that John Donoghue 
shows at his presentations, one of the patients, watching the robotic arm grasp a soda 
can and lift it up under the control of his own mental imagination, stares at the robotic 
hand, moving with his thoughts, and says, in clear amazement, “Whoa.” 


Brain manipulation 

The concept of a separation between brain and mind (dualism) can still survive the 
observation that mental imagery is reflected in neuronal firing, because it could be that 
the brain only reflects what is going on in the mind. The real test of whether the mind 
truly corresponds to neuronal firing in the brain is whether manipulations of that firing 
change experiences. In fact, at the sensory level, this technology is so successful that 
we often don’t realize that it exists. The cochlear implant, a technology implanted in 
hundreds of thousands of patients, stimulates sensory neurons in the ear to translate 
auditory signals from a microphone directly into the brain. 37 Retinal stimulators trans¬ 
lating video images into sensory signals on the retina are currently in Phase II clinical 
trials. 38 

However, clinical stimulation of the deep brain beyond sensory cortices has been 
limited to simple waveforms and frequencies without information. The main clinical 
stimulation of nonsensory structures has been a procedure called deep-brain stimula¬ 
tion, which is currently a common treatment for Parkinson’s disease. 39 Although the 
mechanism by which it works is still unknown, stimulating the basal ganglia at low 
frequencies (under a few hundred cycles per second) often re-enables motion in aki¬ 
netic patients (patients who can’t move). Some researchers think that this stimulation 
effectively shuts off the basal ganglia structures, producing a lesion-like effect, allow¬ 
ing other decision-making systems to take over. Other researchers think that the basal 
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ganglia need to work at a certain rhythm/ and the stimulation enables it to get back into 
that rhythm. Deep-brain stimulation targeted at other areas is now being tried for other 
disorders 41 (such as epilepsy and depression), but most of these experiments are pretty 
much flying blind—no one knows why they are working when they do or why they 
don’t when they don’t. 

Research-wise, however, stimulation has been shown to be able to control every¬ 
thing, including pleasure, pain, and memories. In Chapter 4, we encountered medial 
forebrain bundle stimulation, where some stimulation targets produced euphoric plea¬ 
sure and other stimulation sites did not. Stimulation has been used for everything from 
pain management to muscular control. 42 The motor cortices themselves were defined 
in the 1800s when scientists passed small amounts of current and stimulated the cor¬ 
tex, producing motion in anesthetized animals. 43 Stimulating one location on the motor 
cortex led to arm motion, while stimulation of another led to leg motion. 

In the 1940s, Wilder Penfield and Herbertjasper were trying to help epilepsy patients 
avoid just the sort of disaster that led to H.M.’s memory loss, and examined the effect 
of cortical stimulation in these epilepsy patients. Using local anesthesia to open up a 
small piece of the skull/ Penfield and Jasper stimulated areas of the cortex of awake, 
resting patients, 44 and found that stimulations could trigger memories, hints of songs, 
even smells, and visions. 

The most impressive manipulation of brain signals is still the work by Bill Newsome 
and colleagues from the 1990s, in which they were able to change a monkey’s perception 
enough to change its ultimate decision. 45 Newsome and colleagues trained monkeys to 
observe the flow of random dots on a screen. Some of the dots moved truly randomly 
and some moved in a given direction (say left or right). The monkey’s goal was to deter¬ 
mine which direction the (very small) proportion of dots was moving. We encountered 
this task earlier in our discussion of integrating perceptual signals (Chapter ll). If you 
remember that discussion, one of the visual areas (MST) determines whether the 
whole field is moving together. Because cortical representations in MST are arranged 
in columns, with each column representing a different direction, Newsome and his stu¬ 
dents were able to stimulate single columns in MST and change the perceived direction 
of motion, and the monkey’s responses. Similar experiments have been able to change 
the direction of visual saccades (by stimulating other areas, as well as in subcortical 
structures such as the superior colliculus, which drives orientation of attention). 46 


E There is some evidence, for example, that giving some of these patients a rhythm to work with 
(e.g., through music) can renormalize motion. Sometimes the presence of visual cues is enough to 
re-enable motion. For example, placing a stick across a patient s path can force the patient to step over 
the stick and get started walking again. This is shown very nicely in the movie Awakenings, in which the 
doctor has the floor painted with black and white squares, which allows the patient, “Rose,” to walk to 
the window. 40 

F The brain does not have any local, internal pain sensors, so it is possible to use local anesthesia to 
silence the sensors in the skin and then manipulate the brain of the patient, while the patient is awake 
and on the table. I have seen this done during a deep-brain-stimulation surgery in which deep-brain 
stimulation electrodes were being implanted. The patient was calm and comfortable and chatted with 
the doctors as they implanted the electrodes in his brain. Most lesion surgeries (e.g., for epilepsy) are 
done on awake patients, so that surgeons can determine the role of the structures they are removing to 
make sure they do not remove anything critical (to avoid the tragedy of H.M.). 
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The brain is the physical instantiation of the mind. Interfering with brain function 
changes mental function. One of the reasons people are worried about learning how the 
brain works and learning how the physical brain instantiates the mind is that they are 
worried that we will develop a form of mental control, where machines manipulating 
your brain control your thoughts, desires, and actions. But just as changing the brain 
changes the mind, changing the mind changes the brain. Whenever you learn some¬ 
thing new, your brain is physically changed. When you realize that you really do have 
two choices—to drink or not to drink—that is a physical change in your brain as well 
as a change in your mental state. Humans are already very adept at controlling each 
other—a tremendous amount of our brain power is dedicated to predicting what our 
neighbors, colleagues, friends, and enemies will do. It is said that charm is the ability to 
get people to do something that you want them to do without them realizing that you 
got them to do it. Just because the manipulation is not physical does not mean it is not 
manipulation. The brain is the physical instantiation of the mind. Every mental event is 
reflected in the physical realm, from imagination to hallucination to mental imagery. 
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Addiction 


a falling leaf 
interrupted 

yellow flower petal in my hand 


Addiction is a disorder of decision-making. Defined this way, the 
question of addiction becomes one of identifying the different vul¬ 
nerabilities or failure modes that have arisen within an individual’s 
decision-making systems. This hypothesis suggests that addiction 
treatments need to be tailored to the individual, based on the iden¬ 
tification of those failure modes. Treatment processes are beginning 
to be tailored in just such a way. 


Addiction is the classic disorder of decision-making—an addict continues taking drugs 
even in the face of massive negative consequences, and sometimes even in the case of a 
stated desire to stop. 1 How can an addict both state that he or she wants to stop taking 
drugs and yet continue taking them? As we’ve seen, human decision-making arises from 
a complex interaction of components. What we will see in this chapter is that this com¬ 
plex interacting set of decision-making systems, this decision-making machinery, can 
break down in many different ways. While each of these different vulnerabilities may 
produce subtle differences in how the addict acts ( Does the addict crave the drug before 
taking it? Does the addict take only the one drug, or is any drug good enough? Is the addict 
chasing pleasure or avoiding dysphoria ?), they all come down to the same problem—the 
addict continues the behavior, even though he or she should stop. 


What is addiction? 

Most modern definitions of addiction are based on the concept of maladaptive deci¬ 
sions—continuing to take drugs even in the face of severe negative consequences, 
whether they be illness, dysphoria, or longer-term negatives, such as loss of friends, fam¬ 
ily, job, etc. 2 But this concept can be taken to ludicrous extremes—do we really want to 
say that we are all addicted to breathing? If you take the oxygen away, we die; it s hard to 
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imagine a more negative consequence than that. More problematically, there are many 
cases where we celebrate an individual making a decision with severe negative conse¬ 
quences, such as a soldier defending his or her country, or a firefighter risking his or her 
life in a burning building. We celebrate Martin Luther King, who fought for civil rights, 
knowing full well that it was likely to end in his death. Sometimes, however, this view 
leads to interesting perspectives. The Russian poet Osip Mandelstam continued to write 
poetry even after he had been sent to the Soviet Gulag for it. 3 The future Hall of Fame 
football quarterback Brett Favre continued to play football long after many suggested he 
stop, despite severe injuries and an aging body that was far less resilient to punishment 
than it had been. 4 Should we say that Mandelstam was addicted to writing poetry or that 
Favre was addicted to football? 

Other definitions of addiction have been based on the presence of a disconnect 
between stated desires and actions 5 (“I want to stop, but I can’t”). Of course, this defini¬ 
tion implies that animals (who cannot state such a desire even if they feel it) cannot be 
defined as addicts. Given that animals reliably self-administer the same pharmacological 
substances that humans do, even in the face of negative consequences, 6 this definition 
seems problematic. Another problem with defining addiction as a verbal disconnect is 
that many of us know addicts who do not state a desire to stop their addiction, who 
refuse to admit their “problem.” Many a high school clique has needed to intervene for 
a friend with a drug or alcohol problem, and there are many people alive today because 
their friends stepped in with just such an intervention. 

Some economists have suggested that addiction should be defined by the elastic¬ 
ity of the decision. As discussed in Chapter 3, elasticity is defined by how much your 
decision to buy something changes in response to changes in price. As the economy 
worsens, people tend to go out to the movies or to dinner less often. These luxury items 
are highly elastic. On the other hand, people continue to buy food, even at high prices, 
so food purchases are highly inelastic. Addiction can then be defined as “irrational” 
inelasticity. 7 As noted in Chapter 3, this allows economists to say that Americans are 
“addicted to oil” 8 —we continue to use it and to use it unwisely (driving individual cars 
rather than taking more efficient mass transit; depending on it for electricity and heating 
rather than using non-fossil-fuel systems or taking the time to ensure that our houses 
are energy-efficient), even in the face of severe negative consequences (major oil spills, 
high prices, and unnecessary wars). 

It is important to note that drugs (like oil) are, in fact, economic objects;* as the 
cost of drugs goes up, their use goes down, even among the most dedicated and physi¬ 
ologically dependent addicts. 10 The number of people quitting smoking, for example, is 
directly proportional to the cost of cigarettes. Similarly, studies of changes in heroin or 
cocaine pricing (remember, the black market is still a market) find that increases in pric¬ 
ing lead to direct changes in the number of users and the amount of use. This elasticity is 
why one of the best ways to decrease the use of legal drugs (nicotine, alcohol, and, in the 
near future, marijuana) is to have “sin taxes,” which increase the price of the drug. 

This elasticity is also one of the best arguments against the legalization of harder 
drugs (heroin, cocaine, methamphetamine). The argument for or against legalization is 


A The technical term is “economic good” 9 because one buys “goods and services,” but that would 
lead to the unfortunate sentence “Drugs are, in fact, an economic good....” 
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very complicated. On the one hand, the negative consequences of getting caught with 
drugs is a cost that decreases use. For example, studies of the amount of opium use in 
Taiwan during the Chinese crackdowns in the 1920s find a direct elastic relationship 
between the severity of punishment and initial use. 11 On the other hand, jailing users 
can interfere with treatment because it can reduce the positive options and positive 
interactions that are often closely related to recovery. 12 Jail time without treatment can 
throw users who have not yet committed violent criminal acts in with dangerous crimi¬ 
nals who are, in fact, dangers to society. The other problem with criminalizing drug use 
is that it shifts drugs from an open market to a criminal market and tends to lead to vio¬ 
lent business interactions, which have strong negative consequences for bystanders not 
actually involved in the criminal market. 13 (See Prohibition, US 1920-1933; Columbia 
and the Medellin Cartel 1976-1993; and drug and gang violence in some American 
cities today.) 

In part because of this complexity of defining addiction (and in an attempt to remove 
the stigma that had become attached to the word “addict”), psychiatrists in the DSM-IV 
andICD-10 (regularly released books detailing the currently agreed-upon definitions in 
psychiatry) abandoned the word “addiction” and referred to drug users as “dependent” 
and overuse of drugs as “dependency.” 14 I believe that this definition overemphasizes 
the pharmacological effects of drugs more than the interactions with decision-making. 
It also makes it hard to ask questions about “addiction” to nonpharmacological sub¬ 
stances. I will continue to use the terms “addict” and “addiction” to refer to people mak¬ 
ing these maladaptive decisions to continue using drugs, while recognizing that the 
question of when one is an addict is more of a policy question than a scientific one. 
The scientific question is Why is the person continuing to pursue this drug? How can we 
change that? Whether we want to interfere, and how much we should be willing to push 
to interfere, is a personal question for affected individuals and family members and a 
policy question for society. 


Addiction as a failure of decision-making 

If we look at the decision-making system that we have explored so far in this book, what 
we have is a machinery of decision-making. Like any machine, there are multiple places 
it can break down. Looking at the system in detail, there are several dozen different ways 
that this machinery can make the wrong choice 15 —the Pavlovian system can assign 
increased emotional motivation to a drug, 16 the Deliberative system can misjudge the 
expected outcomes of a choice, 17 the Procedural (habit) system can overvalue 18 or auto¬ 
mate too quickly/ 9 the situation-recognition system can overseparate or overgeneral¬ 
ize situations, 20 etc. Each of these different failure modes or vulnerabilities could have 
an effect of driving the addict to continue making the addictive choice. 21 Differences 
between these failure modes or vulnerabilities imply differences in how the addict makes 
those choices, and each one will likely require different treatments. 

I do not mean to suggest that each addict s behavior will be due to only one failure 
mode in his or her decision-making system. Biology tends to have lots of backup mecha¬ 
nisms. It may well be that full-blown addiction arises only when multiple systems break 
down. Drugs themselves also tend to create additional failure modes. For example, a 
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massively pleasurable effect (the high of cocaine 22 ) leads to increased motivation for 
cocaine (a “desire” for it 23 ), as well as tolerance (requiring more cocaine to produce that 
high 24 ), and an increased likelihood of taking actions leading to cocaine (the Procedural 
system, driving unthinking decision-making 25 ). Alcohol leads to a reduction in anxi¬ 
ety (thus driving a learned need 26 ), tolerance (thus requiring more and more alcohol 
to produce the same effect 27 ), and reductions in the effectiveness of self-control and 
Deliberative systems. 28 Addressing only one vulnerability may not be enough to stop 
the addiction. On the other hand, sometimes one can find a key to allow the patient to 
find another path to prevent relapse. 29 

To understand the concept of a failure mode or a vulnerability in the system, the sim¬ 
plest example is to look at opiate agonists. As we saw, there is good evidence that eupho¬ 
ria (pleasure) and dysphoria (displeasure) are instantiated in the brain by the flow of 
endogenous opioids (endorphin and dynorphin, see Chapter 4). This means that the 
feeling of pleasure is the flow of endorphins onto p-opioid receptors. (This is a direct 
consequence of the rejection of the mind/brain duality that we have been discussing 
in the last few chapters.) Just as stimulating auditory nerves produces hearing sounds, 30 
and waves of random activity across the visual cortex produce images, 31 changing the 
flow of endorphins onto p-opioid receptors changes euphoria signals. This means that 
a pharmacological chemical that tricked the p-opioid receptors into thinking that they 
had seen endorphins would be experienced as euphoric pleasure. What are some chem¬ 
icals that activate the p-opioid receptors? They are the ones that gave us the words opioid 
and opiate, derived from the opium poppy (including such mainstays of addiction as 
morphine and heroin). 32 

Addiction researchers will often talk of the drug as “hijacking” the neural system, but 
I think what is going on is something more subtle. We are taking an engineer s view of the 
brain—there is a physical system and that physical system has ways it can break down. 
We evolved a machinery that includes a “pleasure” component as part of how it processes 
decisions (Chapter 4). This pleasure component is signaled by a chemical pathway. 
A foreign pharmacological substance is changing that chemical pathway. This takes us 
back to the “Car Talk” perspective that we started with in Chapter 2. To return to our 
original example at the start of the book, just as the thermostat in your house would react 
incorrectly if you held a match underneath it (it would think the house was hotter than it 
is), your thermostat would also react incorrectly if you passed a current over the wire that 
signals “I’m hot” to the thermostat. Heroin is changing that internal wire. 

When Steve Jensen, Adam Johnson, and I looked at the decision-making machinery 
in this way in 2008, 33 we found many examples of these sorts of vulnerability points in 
the machinery; in fact, most of those failure points had already been found by other 
researchers. Many researchers had proposed theories of why addicts continue to take 
their drugs (for example, that their system gets out of balance and they need the drugs 
to return to balance; 34 that they are chasing euphoric properties; 35 that they learn to 
overvalue the drugs; 36 that the drugs become secondary reinforcers and draw atten¬ 
tion away from other things; 37 that the drugs shift the balance between deliberative 
and habit systems; 38 etc.). What we found is that these theories were all stated in the 
form of “if you have this problem, you might be an addict.” What was interesting was 
that these theories were generally attacked by a researcher finding an addict with¬ 
out that problem. Logically, what the theories were saying was that a given problem 
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would lead to addiction, not that all addicts had a given problem. Viewed in this 
way, addiction is a symptom that can arise from multiple underlying “diseases.” Our 
multiple-vulnerabilities hypothesis said that everyone was right—all of these problems 
could lead to addiction, because all of these problems were potential failure modes of 
the decision-making system. Figure 18.1 shows a list of some of the current known 
failure modes of the decision-making machinery. 

It is important to note that addiction, by definition, is an interaction between nature 
and nurture. If one never takes cocaine, one never becomes addicted to cocaine and 
cocaine never takes over ones life. Similarly, there are many different reactions to taking 
cocaine; there are many different reactions to smoking cigarettes. 39 Some people remain 
able to control their use, while others are lost to it after one experience with it, and oth¬ 
ers become lost to it only after extended experience. We do not yet know whether this 
is a function of the situations in which these users find themselves, or a function of the 
specific formulation of the drug itself, or the genetics of the user, or the learned back¬ 
ground and life experiences of the user. (Most likely it is a complex combination of all 
of these.) What we do know is that drug addiction presents in many different ways, even 
among the population of people who are unable to control their drug use. 


Craving: The difference between desire, withdrawal, and habit 

One example of this difference is the now-clear clinical dissociation between craving 
and relapse. 40 For years, clinicians and research scientists have studied craving in addicts 
as a substitute for measuring what’s really important—relapse. In large part, they do this 
because craving is something that can be studied in the laboratory, in terms of a con¬ 
trolled, experimental paradigm. For example, in cue-induced craving, pictures of drug 
paraphernalia are shown to admitted drug addicts and to nonaddicts, and the addicts 
show more signs of craving (stress, sweats, nervous behaviors) and rate their feeling 
of craving higher on survey scales. 41 However, it has been known for decades now that 
craving does not imply relapse. While craving and relapse are related (people who show 
a lot of craving are more likely to relapse, and people who relapse are more likely to 
show craving), they do not always co-occur. The problem is that while craving is a phe¬ 
nomenon of a moment, relapse is a long-term event that happens only (as an ethologist 
would say) “in the field.” Before we can actually measure this, we need to address what 
craving is, neurobiologically. 

Craving is the inexorable desire for a thing—we’ve all felt it at one time or another. 
To quote the Rolling Stones, “you can’t always get what you want, but sometimes you 
get what you need.” 42 The mechanism of craving has been studied extensively, particu¬ 
larly in terms of drug addicts being reminded of their addiction, for example by being 
shown pictures of drug paraphernalia. But craving is not such a simple thing. For exam¬ 
ple, it turns out to be dissociable from relapse in drug addiction. Even though someone 
might crave a cigarette or a drug hit, it is possible to resist that craving, and not relapse 
back into the addiction. Others may take the drug-taking action without ever craving it. 
As one patient with a gambling problem told a colleague of mine, “I don’t know what 
happened. I just found myself at the casino.” (Were they lying? Were they lying to them¬ 
selves?) As we’ve seen, there are multiple decision-making systems driving our actions, 
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Failure-point 

key systems 

clinical consequence 

Moving away from homeostasis 

motivation 

withdrawal 

(Koob and Le Moal, 2006; Koob and Volkow, 2010) 


Changing allostatic set-points 

motivation 

physiological needs, craving 

(Koob and Le Moal, 2006; Koob and Volkow, 2010) 


Sensitization of motivation 
(Robinson and Berridge, 2001, 2003) 

motivation 

incorrect action-selection, craving 

Cue-outcome associations elicit 

emotions and 

incorrect action-selection, craving 

pre-wired visceral actions 

motivation 


(Damasio, 1994; Bechara and Damasio, 2002; Bechara, 2005) 


Escape from negative emotions 
(Koob, 2009) 

emotion 

incorrect action-selection 

Mimicking reward 

multiple 

systems 

incorrect action-selection, craving 

(Volkow et at, 2002; Wise, 2005) 



Errors in expected outcomes 

deliberation 

incorrect action-selection 

(Goldman et al., 1999; Jones et al., 2001; Redish 

and Johnson, 2007) 


Increased likelihood of 
retrieving a specific expected 
action-outcome path 
(Redish and Johnson, 2007) 

deliberation 

obsession 

Over-valuation of expected 

deliberative 

incorrect action-selection 

outcomes 



(Redish et al., 2008) 



Over-valuation of learned 

actions 

(Di Chiara, 1999; Redish, 2004) 

habit 

automated, robotic drug-use 

Timing errors 
(Ross, 2008) 

habit 

preferences for unpredictable events 

Over-fast discounting processes 

deliberative, 

habit 

impulsivity 

(Bickel and Marsch, 2001; Bickel and Mueller, 2009) 


Changes in learning rates 

deliberative, 

habit 

excess drug-related cue-associations 

(Franken et al., 2005; Gutkin et al., 2006; Piray et al., 2010; Redish et al., 2008) 

Selective inhibition of the 
deliberative system 

system- 

selection 

fast development of habit learning 

(Bernheim and Rangel, 2004; Bechara, 2005; Bickel and Yi, 2008; Baumeister and Tierney, 2011; Bickel et al., 2012) 

Selective excitation of the 
habit system 

(Everittand Robbins, 2005; Bickel and Yi, 2008) 

system- 

selection 

fast development of habit learning 

Misclassification of situations: 

situation- 

illusion of control, hindsight bias 

overcategorization 
(Redish et al., 2007) 

recognition 


Misclassification of situations: 

situation- 

perseveration in the face of losses 

overgeneralization 
(Redish et al., 2007) 

recognition 



Figure 18.1 Some failure modes of the decision-making system that can lead to 
addiction. Obviously incomplete. 




















Addiction 


ill 


only some of which are conscious. It is likely that if a person were to relapse through 
the Procedural system (Chapter 10), this relapse would not entail any craving before¬ 
hand (or perhaps even any prior recognition beforehand). They might well just find 
themselves “at the casino.” 

An important aspect of craving is that we crave things.* 3 We do not simply crave plea¬ 
sure; we do not simply crave. The word craving is transitive; it must include an object. 
We crave sex, we crave a cigarette, we crave a drink. Perhaps we can even crave love— 
but even then, what we are craving is a feeling, a moment, a romantic dinner, a walk 
on the beach. While that craving might be satisfied by something different from your 
original goal, there is no question that craving includes an identifiable target or goal. 
When someone craves something, there are extensive effects on the body itself. 44 Part 
of craving is that it increases arousal—the heart beats faster and harder, one’s atten¬ 
tion increases, one gets jittery. These are Pavlovian actions that we saw in Chapter 8— 
untrained actions that occur in response to the delivery of the desired object that are 
now occurring in response to delivery of the expectation of that object. 

Somepeople have explicitly identified cravingwith the Pavlovian system—something 
reminds us of the existence of an outcome (through a stimulus-outcome association), 
and we remember that the outcome was very desirable. 45 Just as we discussed when we 
introduced the multiple decision-making systems (Chapter 6), Pavlovian training has 
both action-selection effects (Chapter 8) and motivational effects (Chapter 13). It is 
not clear at this point whether the motivational effects of craving are identical to the 
physiological changes of craving or whether they just tend to co-occur. 46 

Other people have identified craving with the Deliberative system, in that a 
Deliberative system entails a search for an outcome, finding paths that can reach to it. 47 
This means that part of the Deliberative system must maintain a representation of the 
target so that it can be recognized when one finds it. Early studies of craving found that 
craving does not tend to occur for easily achieved goals; instead, it tends to occur for 
blocked goals. Consider an alcoholic who goes to the bar every day at a certain time and 
does not show craving until one day arriving to find the bar closed. Then the craving 
sets in. 48 This blocked path forces the alcoholic from a well-trained, regular habit-based 
(Procedural) decision-making system to the more flexible, more conscious planning 
(Deliberative) system. 49 

When we discussed the motivation system (Chapter 13), we discussed a particular 
phenomenon called Pavlovian-to-instrumental transfer (PIT), in which Pavlovian asso¬ 
ciations reminded one of an outcome, which led to increased actions taken to achieve 
a specific goal. This suggests that craving may be an interaction between the motiva¬ 
tional and Deliberative systems. Experiments directly aimed at PIT suggest a strong 
role for the nucleus accumbens shell, which is strongly involved in craving. 50 A rela¬ 
tionship between PIT and craving would imply that craving is an interaction between 
the Pavlovian motivation (Chapter 13) and Deliberative (Chapter 9) decision-making 
components. 

It is important to dissociate craving from the negative feelings associated with ceas¬ 
ing drug-taking. As quoted from a patient to Anna Rose Childress and her colleagues, 
“No, doc, craving is when you want it—want it so bad you can almost taste it.. .but 
you ain’t sick... sick is, well, sick.” 51 Ceasing drug-taking produces dramatic withdrawal 
symptoms, both physical and mental. Physical withdrawal symptoms come because the 
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drug affects more than just the central nervous system. Biological systems are funda¬ 
mentally chemical systems, and the chemistry of the drug is going to have profound 
effects on the body. Of course, one of the points this book is about is that the mental 
changes are physical as well, and those chemical changes are going to have profound 
effects on both brain and mind. 

Generally, withdrawal symptoms come from homeostatic shifts. The term “homeo¬ 
stasis” means maintenance of something at a given level, from the Greek words opoio 
( homeo -, meaning “same”) and credent; (stasis, meaning “standing still”). Much of biol¬ 
ogy is about maintaining the system at some point that will enable continued survival. 52 
These homeostatic levels are related to the intrinsic goals discussed in Chapter 13 —when 
you need food, you are hungry; when you are too cold, you want to warm yourself up. 
The classic case of homeostasis is body temperature. Just like the thermostat that we 
saw at the very beginning of the book, your body contains temperature sensors (in the 
biological case, neurons throughout that project to a small nucleus of cells in the hypo¬ 
thalamus). If these sensors determine that the body is too hot, processes are initiated 
to cool the body down—capillaries near the skin surface open wider to provide more 
blood flow to the skin, allowing a faster transfer of heat from the internal core to the sur¬ 
face; pores open up, producing sweat, providing evaporative cooling. In extreme cases, 
humans will even open their mouths and pant. Similarly, if these sensors determine that 
the body is too cold, opposite processes are initiated to heat the body up, constricting 
capillaries, pulling the skin tighter (producing goose bumps), exercising muscles, etc. 53 

These days, however, a number of scientists have recommended replacing the term 
with “allostasis,” from the Greek word aXXo- (“alio-,” meaning “other”). 54 In part, this 
redefinition is due to the implication that in homeostasis one maintains the parame¬ 
ter at a single target, while, in actuality, the target changes with ones needs. (We used 
to hear that a normal human body temperature is 98.6 degrees Fahrenheit, but body 
temperature actually changes throughout the day and, of course, changes in fever. 55 ) In 
addition, biological systems learn to plan—if you are in a world where food is scarce, 
it’s better to overeat when food is available than to eat just until you’re full, because 
you don’t know if you’ll be starving later. 56 Part of the role of the Pavlovian motivation 
system (Chapter 13) is to guide that plan. 

So what does this have to do with withdrawal symptoms? The key is that homeo¬ 
static systems react slowly. Imagine that your body is trying to maintain a certain oxygen 
flow through the bloodstream. If you move to a high altitude (say Katmandu in the 
Nepalese Himalayas), then getting enough oxygen to your tissues requires your body to 
put more resources into hemoglobin and oxygen transfer. This happens over the course 
of days or longer. Similarly, if you then move back to a lower altitude, the extra oxygen 
flow remains for a couple of weeks while your system readjusts. Athletes who require 
high oxygen flow (such as Olympic long-distance runners) often live or train at high 
altitudes so that their body will adjust and be better at delivering oxygen to their tissues 
when they run the actual race (although the efficacy of this is still controversial). 57 

A cigarette smoker has been flooding his or her tissues with nicotine, which is an ana¬ 
log of a very common brain chemical called acetylcholine. 58 The music that I am listen¬ 
ing to right now on my computer has three volume knobs in sequence. The media player 
has a volume, which changes how loud the music is that is sent to the computer, which 
has a system volume, which changes how loud the music is that is sent to the speakers, 
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which have their own volume. If the media player volume increases, then I can get the 
same level of music out by decreasing the volume on the speakers (which is easier to 
reach than the internal system volume). But then if the media player volume suddenly 
decreases, the music will be too soft, and I won’t be able to hear it. 

In response to the chemical flooding of nicotine, the brain has learned to turn the 
volume down on acetylcholine receptors. So when a person stops smoking, the body 
and brain react as if they weren’t getting enough acetylcholine—because the ex-smoker 
isn’t getting enough acetylcholine, because the volume on the acetylcholine receptors 
has been turned down too low. Such changes occur throughout the brain and peripheral 
systems. Withdrawal symptoms typically occur for a time after stopping drug use (days, 
weeks, months). These withdrawal symptoms are due to the body’s adjustment being 
out of alignment with the drug expectation. 59 Of course, as we discussed above, it is pos¬ 
sible to change the body’s homeostatic (allostatic) target based on cues and situations. 
This means that cues can remind an addict of an expectation of drug, leading to a shift in 
the allostatic target, leading to what is effectively cue-induced withdrawal symptoms. 60 

But craving can also occur in the absence of any explicit withdrawal symptoms. 61 Craving 
is a mental process due to more permanent changes in learning and memory systems and 
can arise long after the withdrawal symptoms have gone away. Even though craving occurs 
through a physical process (everything in the brain is a physical process), understanding 
what craving is physically has been much more elusive and will likely depend on under¬ 
standing how craving interacts with the multiple decision-making systems. 

The key experiment would be to measure craving continuously as people live their 
lives, and then to compare those who report high craving levels and those who don’t 
and to compare those who relapsed and those who didn’t. This is, of course, a very 
hard experiment to do. There have been some attempts using cell-phones and PDAs 
to ask people to report craving at regular intervals. 62 These experiments generally find 
that craving does often occur before relapse, but that sometimes craving occurs with¬ 
out relapse and sometimes relapse occurs without craving. Many readers may be ask¬ 
ing why we don’t use retrospective experiments that find people who relapsed and ask 
them whether they showed craving beforehand. The problem comes back to the recon¬ 
struction of memory that we saw in the last few chapters. We don’t actually remember 
the episodes of our past; instead we reconstruct them from our memories of what was, 
what should have been, and what probably happened. In fact, careful experiments have 
found that people who relapse without craving often remember (incorrectly) that they 
showed craving before the relapse. 63 

Craving can occur in an attempt to alleviate withdrawal symptoms. 64 Craving can 
occur because the emotional (Pavlovian, Chapter 8) system is expecting a strong reward 
(or needs to avoid a strong punishment), 65 or because the planning (Deliberative, 
Chapter 9) system recognizes a path to a strong reward. 66 It is not yet known whether 
these different drives will be dissociable experimentally or not, but they all entail mech¬ 
anisms by which a person would express a strong desire for the drug. 

Interestingly, all three of these mechanisms may lead to craving, but they don’t nec¬ 
essarily lead to relapse. The separation between craving and relapse comes back to the 
multiple decision-making systems that we have seen throughout this book. Craving is a 
function of the expectation of the outcome; you can’t crave something you don’t know 
you’re going to get. This means that craving won’t occur when actions are selected by 
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the Procedural system (Chapter 10), which doesn’t include a representation of the out¬ 
come. Since the Deliberative system implies recognition of a path to a goal, but knowing 
that path does not mean one takes it (remember, the Deliberative system is flexible), one 
can have craving without relapse. Since both reflexes and the Pavlovian system can be 
overridden by self-control systems (Chapter 15), they don’t necessarily lead to relapse 
either. While craving and relapse are related, they are not identical. 


Paths out of drug use 

Although drug users, scientists, and society would very much like to have a sin¬ 
gle magic bullet that would “cure” a drug user of the predilection for drugs, the 
multiple-vulnerabilities theory of drug addiction suggests that there isn’t going to be a 
single path out of addiction. 67 That being said, current practice is to try multiple options, 
and some subset of these options often works for specific individuals. 68 My suspicion is 
that we need to see drug addiction not as the problem in itself, but rather as a symptom 
of an underlying problem or set of problems.® 

Although this book is explicitly not a self-help book, and I am not going to attempt to 
tell a drug user how to escape from his or herpredicament, c the multiple decision-making 
story that we have been telling throughout this book has some interesting implications 
for identifying potential paths out of the cycle of drug use. 

The implication of this engineer’s view on addiction and the concept that there are 
multiple decision-making systems with multiple vulnerabilities provide two important 
perspectives on treatment that can be addressed. First, the effect of treatments should 
be to normalize vulnerabilities. It should be possible to predict treatment success from 
identification of outcomes of successful and unsuccessful treatment attempts. Second, 
different vulnerabilities will likely require different treatments. If we could identify 
which vulnerabilities an addict has fallen victim to, we could potentially provide the 
right treatment for those vulnerabilities. 73 


B This view, that psychiatric categorizations (drug dependence, schizophrenia, depression, etc.) are 
actually categories of symptoms rather than reflections of an underlying disease, is related to the concept 
of endophenotypes. The term “endophenotype” comes from the idea that different genes can have the 
same effects (phenotypes), hut the concept here goes beyond simple statements of genetic variation. 
The concept is closer to that of hypothetical constructs or trans-disease processes. 69 Hypothetical constructs 
are intervening variables that allow one to look for underlying concepts in neural activity (such as sub¬ 
jective value, 70 which we discussed in Chapter 3), while trans-disease processes are treatable vulnerabili¬ 
ties that produce different dysfunctions in different contexts (such as impulsivity, 71 Chapter 5). 

It is true that psychiatric categories have become reliable over the past decades such that they can 
be consistently applied—multiple psychiatrists will generally assign the same people to the same cat¬ 
egories. However, successful treatment has lagged behind. Unfortunately, category reliability does not 
necessarily imply a unified mechanistic validity. 72 There is a current movement afoot called computa¬ 
tional psychiatry that attempts to redefine psychiatric illnesses in terms of their underlying mechanisms, 
connecting psychiatric symptoms with underlying dysfunction based on computational models of 
decision-making. We will return to this concept in Chapter 21. 

c As noted in the preface, treatment needs to be tailored to the individual. Thus, one should check 
with one’s own medical professionals before embarking on any treatment regimen. 
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Some evidence exists suggesting that these perspectives are correct—different 
addicts are better served by different treatments. For example, behavioral impulsivity 
measures predict success in smoking cessation. Neural structures and cognitive tests can 
be identified that predict treatment outcome success in cocaine addiction. 74 

Some vulnerabilities will be simple to treat. A homeostatic imbalance can be allevi¬ 
ated by replacement with reduced-harm alternatives, such as nicotine replacement ther¬ 
apy for smoking or methadone replacement for heroin. 75 However, long-term relapse 
rates after ceasing these treatments are notoriously high. 6 This is likely a consequence of 
the fact that these therapies do not address the other vulnerabilities that an addict may 
have fallen victim to. 77 

For example, it is important not to underestimate the negative component of drug 
addiction on emotion and affect. Because of the way that biological systems have lim¬ 
ited internal resources, and because biological properties are always trying to balance 
themselves, a biological high (euphoria) will be followed by a crash (dysphoria). 78 
If euphoria, for example, is the release of endorphins onto p-opioid receptors, then her¬ 
oin flooding the system will produce an overcompensating reaction like “ringing” in an 
oscillator. These negative effects can occur after even a single dose of morphine. 79 And, 
as we saw above in our discussion of homeostatic (allostatic) systems, these systems will 
learn to compensate for that overabundance of signal, leading to a negative assessment 
in contrast, when everything else just feels terrible. This is shown clearly in the movies 
Sid and Nancy and Trainspotting: the addicts’ senses have become so dulled to every¬ 
thing that drugs are the only thing left that can hold their interest for even a moment. 
Finding healthier activities that can substitute for drugs or alternative emotional factors 
that can capture one’s desires can be difficult but potentially useful. Sometimes the key 
is recognizing just how bad off one is and that alternatives do exist. 80 

Some users have been able to break their addiction through attention to concrete 
alternatives, whether those alternatives are social (such as attending regular social inter¬ 
actions where one is praised for rejecting the addiction) or physical (such as earning 
money or points toward a new television set). 81 There is good evidence now that concrete 
targets are easier to aim toward than abstract ones and that concrete targets draw more 
of one’s attention. 82 When those concrete targets are immediately-available things one 
wants to reject, they are more difficult to reject. When those concrete targets are positive 
goals in the future, they are easier to attend to and easier to assign value to (thus making 
them easier to use as alternatives). I know a person who broke his smoking addiction by 
putting the money he usually spent on cigarettes in a jar every day. He was a jazz fan and 
set himself a goal of buying a new saxophone. When enough money had accumulated in 
the jar, he had quit his smoking habit and celebrated by buying himself the saxophone. 

Some vulnerabilities can be exceptionally difficult to treat. As we saw in Chapter 10, 
stimulus-response decisions can be made without conscious attention or thought, 
and such habits can be very hard to break. We examined one potential solution in 
self-control, in that one can override those actions with frontal cortical self-control 
mechanisms (Chapter 15). However, those mechanisms require constant attention and 
are very tiring. Distraction, exhaustion, and stress can all make maintaining self-control 
very difficult. 83 Another alternative is to identify what cues lead to relapse and to avoid 
those cues. For example, an alcoholic might drive a different path home to avoid driving 
by the bar where he usually stops in for a drink. 
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This leads us to the concept of precommitment and bundling . 84 Imagine an alcoholic 
who knows that he will drink if he goes to a party that evening, but would prefer to 
stay sober when asked in the light of day. This alcoholic is, in a sense, two people—in 
the light of the day, sober and sane, and in the Pavlovian context of the party, a wild 
alcoholic. (This dichotomy of choice is, of course, the central theme of Robert Louis 
Stevenson’s Dr. jekyll and Mr. Hyde.) These two temporally separated people are in con¬ 
flict with each other. 85 If the sober and sane person in the light of the day (let us call him 
Dr. Jekyll) wishes to stay sober through the night, he only has to avoid the party. The 
problem is that in the evening, Dr. Jekyll will have changed his mind and will want to 
go to the party. One option is to find a way to commit himself to the nondrug option 
during the light of the day (for example, by making other plans for that evening). This is 
called precommitment because Dr. Jekyll has precommitted himself to an option so that 
the later option is not available. 

Bundling is a similar process through which one changes the set of available out¬ 
comes by rethinking what the potential outcomes are. 86 For example, the concept “there 
is no such thing as one drink for an alcoholic” implies that one does not have the option 
to go to the bar and have “just one drink.” 87 Instead, the options are to not drink and 
be the boring Dr. Jekyll or to drink and become the devil Mr. Hyde. Being the cool 
Mr. Suave drinking one drink and charming the ladies is not an option for our pro¬ 
tagonist Dr. Jekyll. Recognizing that there are only the two options (boring Dr. Jekyll 
and dangerous Mr. Hyde) leads our protagonist to a very different decision process 
(between the two options of Jekyll and Hyde rather than between the three options of 
Jekyll, Hyde, and trying to be Mr. Suave). Bundling and precommitment are two exam¬ 
ples of ways that changing ones perspective can change decision-making. 

One interesting issue is that many (though unfortunately not alP) addicts age out 
of their addiction. 88 Whether this is due to changes in brain function with age (addic¬ 
tion often starts in adolescence, when prefrontal self-control functions are impaired 89 ) 
or whether this is due to changes in life (such as gaining a family) is still unknown (and 
probably differs from person to person). Nevertheless, it is interesting that case studies 
of addicts who do age out usually talk of changes in perspective that lead to a rejection 
of the addictive choices. 90 

There are too many treatment options available and their successes and failures 
are too complex to list here or to identify how each one interacts with the multiple 
decision-making systems. As noted above, different treatment options seem to work for 
different patients. 91 Current treatments are now suggesting that patients go through a 
flowchart of options—if the patient has impulsivity problems, this can be treated with 
executive function therapy (which decreases impulsivity); if the patient has a homeo¬ 
static problem, this can be treated with pharmacological replacement. 92 Other problems 
require other treatments. 

Although there are some examples where treatments aimed at specific dysfunctions 
are identifiably successful, 93 a vulnerability-by-vulnerability treatment regimen is as 
yet unavailable. How to identify which failure mode is active (or which failure modes 
are active) in a given addict and how to treat those failure modes are a focus of active 
research throughout the neuroscientific and medical communities today. 94 


Some addicts die before getting a chance to age out and others never stop their addiction. 
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Gambling and Behavioral Addictions 


Driving to the city of illusions, 
the cars ahead disappear 
into the mirage at the point of 
perspective. 

In the shimmering desert, 
pyramids and towers scrape the sky. 


From theperspediveof addiction as dysfunctional decision-making, 
gambling and other behaviors can also be seen as "addictions." 
These nompharmacological addictions tend to depend on vul¬ 
nerabilities in the support systems (e.g., motivational systems, 
situation-recognition systems) more than in the action-selection 
systems themselves, but they do also depend, of course, on how 
action-selection systems use those support systems. 


An interesting question that has arisen over the past few years is whether addiction is 
fundamentally related to drug use or whether one can be addicted to something that is 
not drug-related. 1 From the old “dependence” theory of addiction—that drugs change 
your internal balance of some chemical 2 —it is very hard to describe nondrugs as an 
addiction. But from the “maladaptive” theory (that you put up with high costs to get 
your hit) or the similar economic “inelastic” theory (that use doesn’t decrease as much 
as one expects when costs increase), 3 there are lots of things that one can get addicted to. 
But there are also lots of things that are inelastic that we don’t want to say are bad (like 
breathing!). However, working from the “malfunctioning decision-making machinery” 
theory described in this book, we can see that behaviors too can become overselected 
given the underlying correct valuation they should have. 

In the clinical world, the most common non-drug-related addiction is problem 
gambling, which affects between 1% and 4% of the U.S. population, depending on the 
specific criteria used to define it. 4 Problem gambling is usually identified because the 
maladaptive consequences (losing money, family, stability of life) are often so clear. 
From the decision-making perspective, problem gambling is definitely an addiction. Our 
decision-making system is not optimal; it can be fooled into making poor decisions under 
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the wrong conditions. 5 As with our description of multiple potential ways that pharma¬ 
cological substances (drugs) can access the decision-making system and can drive incor¬ 
rect decisions, there are multiple ways that sequences of wins and losses (gambling) can 
drive incorrect decision-making. We discussed some of these potential vulnerabilities in 
the last chapter, which concentrated on the pharmacological addictions. 

In addition to these motivational effects, humans are very poor at recognizing ran¬ 
dom sequences as random and are always looking for underlying causes. 6 This means 
that we can be fooled by limited experience with a probabilistic event. Because we do not 
see the correct distribution of possibilities, or the correct distribution of the likelihood 
of those possibilities happening, we think we have more control over the system than 
we actually do. (This is most likely a failure in the situation-recognition system 7 —we are 
unwilling to accept coincidence and are always constructing narratives to explain why 
things turned out the way they did. 8 ) 

In particular, humans are very susceptible to the “near-miss,” 9 where we are told that 
we “almost got it,” which leads us to think that we are making progress on figuring out the 
complex system that is the world in which we live. This is something that slot-machine 
makers have learned to exploit. 10 Over the past several decades, slot machines have got¬ 
ten more and more complex until they are almost like videogames these days. Of course, 
casinos have also learned how to reduce one’s ability to recognize these complexities by 
providing an abundance of lights and cues all around the casino to draw your attention 
away, and by providing alcohol to diminish your ability to plan and think. 


The illusion of control 

Imagine taking the 10 digits (0 through 9) and ordering them. Which sequence of num¬ 
bers seems more random to you: 4037981562 or 0123456789? I’ll bet it’s the former 
(unless that’s your Social Security Number). Actually, they are each equally likely. If you 
are selecting a random sequence of 10 digits (without replacement), the probability of 
seeing any specific sequence is 1 in 3.5 million. A 

Several researchers have argued that the reason animals evolved brains is prediction: 
the better you can predict what’s going to happen, the better you can prepare for it, the 
better you can react to it, the better you are going to be at surviving and procreating your 
genes into the next generation. 11 We are evolved to recognize patterns, even if they arise 
from true randomness. This has several effects on our decision-making abilities. First, it 
means that we are terrible at generating random sequences. 12 Second, it means that we 
find patterns even in randomness. 13 


Generating randomness—you just can’t do it 

Ask a hundred people to generate a string of five random digits and I’ll bet none of 
them will say “00000” or “99999” (Unless they’re neuroscientists or psychologists 


A With 10 digits to choose from there are 10 possibilities for the first digit, 9 for the second (all but 
the one you already chose), 8 for the third, etc. This works out to be 10X9X8X7X6X5X4X3X 
2x1 = 10! = 3,628,800. All possibilities, including 9876543210, are equally likely. 
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who study this stuff and will come up with those numbers just to screw with your 
experiment—but of course, then those numbers aren’t random either.) In general, 
people will start off with a number (say 7) and then try to find a number that is dif¬ 
ferent from that, and then come back to another number. This is a process called the 
“inhibition of return” and is suggested to be a means of improving foraging. 14 In a sense, 
one is “foraging” for numbers, 15 and, having found 7, one wants to find the next num¬ 
ber elsewhere. 

My favorite demonstration that randomness is hard to do is that computer programs 
can learn to beat us at rock-paper-scissors. 16 This is the game where you and your oppo¬ 
nent count to three and, on three, place your hand as a fist (rock), as a flat palm (paper), 
or with two fingers making a cutting motion (scissors). In this game, each choice beats 
one of the other choices and each loses to one choice in a ring: rock beats scissors, 
which beats paper, which beats rock. If you can predict what your opponent is going to 
do, then you can always win. But if you can’t predict, then the optimal strategy is to play 
randomly. Unfortunately, humans are terrible at playing randomly. As part of his gradu¬ 
ate thesis work, Steve Jensen at the University ofMinnesota developed an algorithm that 
basically remembers sequences that it has observed. It uses these sequences to predict 
future outcomes. Because it has a very large memory, this program will eventually beat a 
human opponent. If it plays only one game against you, then who wins will be random. 
But if you play a hundred games against it, it will win more than half the time (a lot more 
than half!). It’s a remarkably frustrating experience, because even if you know you have 
to play randomly and you try to play randomly, the computer keeps guessing correctly 
what you’re going to do! 

Predicting what your opponent is going to do is a key factor in most competitive 
sports, whether it be a baseball batter predicting which pitch the pitcher is going to 
throw (and where he is going to throw it), a football coach predicting whether a pass 
play or a running play will be coming up next, or a tennis player predicting where her 
opponent will serve the ball next. Even a small improvement in knowing the next option 
will play out big over the course of a match or a season. And, of course, the ability to 
generate randomness (or its converse, the ability to predict what an opponent is going 
to do) has important implications for other pursuits as well, including business, politics, 
and war.® 


Recognizing patterns 

The other implication of our predilection for prediction is that we recognize patterns 
that are not there. 17 From the man in the moon to religious icons on moldy bread to 


B This doesn’t mean that being random is the best strategy to win in any of these pursuits, whether 
it he football, business, politics, or war. Because some options are actually better than others and 
because interactions between people are rarely zero-sum, true randomness is rarely an optimal strat¬ 
egy. (Rock-paper-scissors is one of the few cases where random is the optimal strategy.) Because we 
are social creatures, two cooperating players are going to be much stronger than two random players. 
Randomness rarely wins you the game, but it often takes down your nearby opponents as well. We’ll 
come back to the issue of cooperation in our discussion of morality (Chapter 23). Suffice to say for now 
that cooperation also depends on the ability to predict what your colleagues are going to do. 
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patterns in marble tiles, our brains look for patterns in the world. Some of these patterns 
are real. Mushrooms and flowers tend to appear after a big rainstorm. A fist coming at 
your face is likely to hit you if you don’t block it and likely to hurt if it hits you. Other pat¬ 
terns are not real. The man in the moon is an illusion caused by an interaction between 
the specific pattern of dark lowlands and bright highlands on the moon’s surface with 
the facial-recognition systems genetically hardwired into our brains. 

The recognition of patterns that don’t exist becomes a real problem when it leads to 
the illusion of control. 18 In a casino, playing a game, sometimes you win and sometimes 
you lose. Let us assume, for now, that these games are truly random. But imagine that 
you notice that you broke a long losing streak just after you touched your hat and wiped 
your brow. Maybe that’s worth trying again. Of course, it probably won’t work the sec¬ 
ond time, but now we’re in a probabilistic world—trying to figure out whether winning 
is just a little bit more likely in one condition or another. 

Of course, logically, it doesn’t make sense why wiping your brow would change a 
slot machine’s outcome, but what if, instead, we think of how you push the button, or 
when you push the button in the course of a spin of some video effect on the machine? 
Now, we have a potential logical cause that could affect the machine’s outcome. Imagine 
a slot machine with a simple switch. The switch could even have a sign on it that says 
“this switch does nothing.” Yet someone playing the machine will likely try the switch, 
and will be more likely to win with the switch in one position or the other. (Although 
flipping a coin will come up tails half the time on average, the chance is very low that 
any specific sequence of coin flips will be half heads and half tails. Similarly, the number 
of wins you observe with the switch in each position is very unlikely to be exactly the 
same.) So, the player will believe (incorrectly) that the switch has an effect. This is the 
illusion of control. 

This is effectively a form of superstition. 19 B. F. Skinner (whom we met as a promi¬ 
nent behaviorist in our discussion of reinforcement in Chapter 4) showed that pigeons 
would eventually do interesting, individual random behaviors if they had done them a 
couple of times before getting a reward, even if the reward was actually being delivered 
randomly. 

A large part of our intelligence is dedicated to trying to predict the world. Several 
researchers have argued that learning is about recognizing contingencies —whether an 
event is more (or less) likely to occur in the context of another event. 20 As we saw in 
our discussion of situation-recognition and the construction of narratives (Chapter 12), 
much of our cognitive effort is spent on recognizing causal structure within the world. 
This has been a hypothesis for the origin of mythologizing 21 since Giambatista Vico. c 

So what happens when your causal hypotheses make mistakes—when you recog¬ 
nize differences between two situations that are not really different? This leads to the 
illusion of control—if you think that there are two situations (one in which you win 
and one in which you lose) and you can find a way to get yourself into the winning situ¬ 
ation, then gambling makes sense. Of course, this is where cognitive errors can lead to 
problem gambling. 23 Think of the stereotypical gambler in the movies—“I know what’s 


c Giambatista Vico was an Italian scientist and philosopher who lived from 1668 to 1744 (just after 
Newton and Descartes). Vico is best known for his Scienza Nuovo (New Science), which is one of the 
main sources for the structure of James Joyce’s Finnegans Wake. 11 
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different. Loan me the money; it’s a sure thing this time. I’ll pay you back when I win.” 
That gambler isn’t hooked on randomness. That gambler isn’t looking to play the game. 
That gambler thinks he understands the causal structure of the world well enough to 
control it. That gambler has succumbed to the illusion of control. 

We can also see that incorrectly recognizing when a situation has changed can lead 
to gambling problems. If a situation has changed, but you do not recognize the change, 
then you will be performing the wrong actions in the wrong situations. Again, cognitive 
errors lead to incorrect decision-making. Here, again, we can see this effect best in clas¬ 
sic literature of tragic inabilities to recognize when the world has changed—an athlete 
who no longer has the abilities of his youth but can’t quit, the businessman unable to 
recognize that the new competitors are using different tactics, the generals fighting the 
last war. 24 

This is something that casino operators and slot-machine makers understand. Over 
the past several decades, slot machines have become more complicated, not less. 25 In her 
book Addiction by Design, Natasha Dow Schiill describes slot and video machine gam¬ 
blers as succumbing to a particularly procedural deficit, getting drawn in to the “flow” of 
a Procedural system, and describes how slot and video game designers have focused the 
machines to access that procedural (non-Deliberative) learning system. Video gambling 
machines include more and more complex decisions and controls, providing more of a 
“game-like” experience, and are less and less about simply putting money in and pull¬ 
ing the lever. More complex options provide more potential options for the illusion of 
control to grab hold of. 26 

In a classic study from 1975, Ellen Langer and Jane Roth trained 90 Stanford under¬ 
graduate students to play a coin-flip game. 27 Students who guessed correctly early were 
much more likely to decide that they were correctly predicting the outcomes than stu¬ 
dents who guessed incorrectly early. That is, a sequence of wins followed by losses led 
the students to imagine they understood the task and needed to try to figure out what 
was going wrong, while a series of losses intermixed with wins led the students to think 
it was random. A sequence of four correct early guesses was enough to lead people to 
believe they could correctly guess the sequence “even for sophisticated subjects.” This 
is how the casinos hook people—a quick series of wins leads you to think that you can 
beat the game, even though (as we’ve seen) WWWWWLLLLL is just as random as 
WLWLLLWWLW. 28 


Wins and losses 

Another potential breakdown is that wins and losses are represented differently in 
the decision-making system. Weve already seen how people value wins and losses 
differently. 0 We’ve already seen how the differences between punishment and disap¬ 
pointment imply that there must be different signals for wins (gains, rewards) and losses 


D You may remember the framing examples that we saw in Chapter 3, where phrasing the exact 
same problem in terms of gains or losses changed which option someone preferred. In the Asian flu 
example, being certain to save 200 lives (out of 600, thus losing 400) was preferred against the gamble 
of saving or losing everyone, but the gamble was preferred over being certain to lose 400 lives. 29 
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(punishments), because disappointment (lack of delivery of an expected gain) is not a 
simple punishment, and relief (lack of a delivery of an expected or potential punish¬ 
ment) is not a simple win (see Chapter 4). 

An interesting example of this can be found in Parkinsons disease, which occurs 
when a subject has lost his or her dopamine cells, particularly those in the dorsal aspects 
of the dopamine areas of the midbrain called the substantia nigra. 30 These are, in fact, 
the dopamine cells that we’ve seen train up the Procedural system (Chapters 4 and 10). 
One of the primary treatments for Parkinsons disease is to increase the amount of dop¬ 
amine in the system, to allow the surviving dopamine cells to work harder (usually by 
providing a dopamine precursor such as levodopa). 31 This can cause a problem because 
Parkinson’s patients still have some of their dopamine cells intact, particularly those in 
more ventral aspects, which seem to be involved in the more cognitive, deliberative, and 
impulsive (particularly the emotional and evaluative) decision-making systems, lead¬ 
ing to impulsivity and manic, emotional problems. 32 This is what happens to Leonard 
in the movie Awakenings . 33 Leonard’s Parkinsonianism is cured by levodopa, but over 
time, the dopamine builds up and Leonard becomes psychotic and his treatment has 
to be stopped. 

If these dopamine cells are involved in learning, then Parkinson’s patients should have 
trouble learning new things from rewards. Both Michael Frank and Mark Gluck have 
run separate experiments training patients to learn from rewarding or punishing signals. 34 
They both find that normal subjects can learn from positive examples (get rewards if they 
choose the correct thing, get nothing if they choose wrong) and from negative examples 
(lose nothing if they choose the correct thing, lose something [usually money] if they 
choose wrong), but Parkinson’s patients have difficulty learning from the positive exam¬ 
ples (rewards). They can, however, learn from the negative examples (punishments). If 
the key to learning is actually dopamine, then one might expect this to reverse under 
levodopa treatment—with levodopa boosting the overall dopamine levels, one might 
predict that it would be possible for Parkinson’s patients to learn from positive rewards, 
but no longer from negative punishments. That is, in fact, exactly what happens. 

As noted above, slot machines and other casino games have increased the number of 
near-misses (I almost won!) that turn a true loss into a sensed win. 35 If part of the effect 
of these drugs is to increase a patient’s sensitivity to rewards while decreasing the sen¬ 
sitivity to losses, one would expect Parkinson’s patients on levodopa and other similar 
drugs that increase dopamine’s efficacy to become highly susceptible to impulse disor¬ 
ders such as problem gambling, where they learn from the positive examples (I won!) 
and not from the negative (I lost). That is exactly what happens. 36 


Other behavioral addictions 


There have also been descriptions of people “addicted” to a number of behaviors, such 
as food, sex, porn, shopping, video games, and even surfing the Internet. 37;E A number 


E What people tend to get addicted to when surfing the Internet are things like gambling, porn, 
videogames, and shopping, so it is not clear whether the problem is the Internet per se or if the Internet 
is just a new medium enabling other behavioral addictions. 
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of these behaviors have been shown to release dopamine, which is often referred to 
(unfortunately in the scientific literature as well as the popular news media) as the key 
to addiction. (Remember that dopamine is one access point, but not the only one, 
to addiction. 38 See Chapter 18.) Even video games lead to the release of dopamine in 
player’s brains. 39 But, as we’ve seen, this is exactly what should happen in the normal 
learning system (Chapter 4), which uses dopamine to learn appropriate behavior. The 
question (the answer to which is not known) is whether that dopamine signal becomes 
correctly compensated as you learn the game. 

For example, is obesity an addiction? There are some data showing that obese people 
have some of the homeostatic compensation and other dysfunctional dopamine mecha¬ 
nisms seen in long-term drug addicts. 40 Some of the data suggest that the problem with 
obesity is related to the high fat and sugar content of junk food. Remember that sugar 
may well have some of the misattribution problems we’ve seen in addictive drugs. 41 So 
it is not completely out of the realm of possibilities that obesity is a sign of a form of 
food addiction, but it is still an open question being discussed in the neuroscientific and 
medical literatures. 

Many of these behaviors are internally driven needs and desires that get out of 
control. As we saw in our discussion of intrinsic reward functions (Chapter 13), evo¬ 
lution only requires that the behaviors have the correct outcome, not that they be “for 
the right reason.” There are a number of behaviors that have evolved to be pleasur¬ 
able for themselves, but were evolved because they were really useful for their conse¬ 
quences in the environment they were evolved in. The classic example is, of course, 
sex. The reason that sex is necessary evolutionarily is, of course, to produce the next 
generation. But the reason that our brains want to have sex is because it feels good and 
it satisfies an internally driven need, not because it produces children. Dogs love to 
chase things (cars, sticks, squirrels). In the wild, dogs were pack hunters that chased 
down big game. So dogs evolved to like to chase things. The chase itself became plea¬ 
surable. The human male’s propensity for porn is presumably a similar effect. It can 
be easily seen how this can lead to overselection of these behaviors in ways that can 
interfere with one’s life. 

So do we want to say that one is “addicted” to these behaviors? What about other 
behaviors, such as shopping or setting fires? 42 Are these behaviors one can become 
addicted to? From the decision-making perspective, the term “addicted” is not the 
important scientific question. From the decision-making perspective, the scientific 
question is why the person is continuing to do the behavior, whether that behavior is 
taking drugs, gambling, or watching porn. Fundamentally, whether or not we should 
call that person “addicted” is a policy question, which determines whether we should 
encourage or discourage that behavior, and how much we are willing to commit to that 
policy. But if we understand how the decision-making process is working to drive the 
person to continue the behavior, we can understand better what one needs to do to 
strengthen or weaken, to enable or disable, to encourage or discourage it. 

Books and papers for further reading 

• Constance Holden (2001). “Behavioral” addictions: Do they exist? Science, 294, 

980-982. 
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• Natasha Dow Schiill (2012). Addiction by Design: Machine Gambling in Las Vegas. 
Princeton, NJ: Princeton University Press. 

• Mark G. Dickerson and John O’Connor (2006). Gambling as an Addictive Behavior. 
Cambridge, MA: Cambridge University Press. 

• Willem Albert Wagenaar (1998). Paradoxes of Gambling Behavior. Hillsdale, NJ: 
Lawrence Erlbaum Associates. 
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Post-Traumatic Stress Disorder (PTSD)* 


water pooling on the floor 
shards of glass remember 
a bright red rose 


In PTSD, devastating memories interact with vulnerable neural 
structures to produce an overwhelming emotional, episodic mem¬ 
ory of an experience. The multiple decision-making system theory 
and its interacting support structures suggest a perspective on 
PTSD as interacting episodic and emotional memories. 


After World War I, soldiers returning from the trenches of Europe were described as suc¬ 
cumbing to “shell shock”: jumpy, cowering at the slightest noise, often unable to func¬ 
tion in normal society. In World War II, the term was “battle fatigue.” The classic image 
is that of a father in 1950, sitting at the kitchen table, smoking his cigarette, staring out 
into Iris memory. This difficulty that some soldiers have is not a new phenomenon. 1 In 
his description of soldiers coming back from the American Civil War, Jacob Mendes Da 
Costa described a phenomenon of sudden-onset heart palpitations, which he called “irri¬ 
table heart” and came to be known colloquially as “soldier s heart.” 2 A 2006 historical study 
found that the best predictor of soldier s heart in Civil War veterans was the percentage of 
the company killed, presumably a reliable proxy for experiencing devastating emotional 
trauma. 3 The term used now, post-traumatic stress disorder, recognizes that this problem 
can arise from traumatic non-military experiences as well: victims of violent crime, sexual 
assault, terrorist attack, car crashes, and other disasters are also vulnerable to PTSD. 4 

At this point, it is not clear what the cause of PTSD is. The symptoms, however, are 
consistent and recognizable (as can be seen in the similarities of the descriptions of 
shell shock, battle fatigue, and soldier’s heart). 5 These symptoms include uncontrollable 
re-experiences of the event. Based on our separation of episodic and semantic memories 
(Chapter 16), these re-experiences would be unquestionably episodic in nature. They 
are often vividly present and intrusive. Patients with PTSD also show a withdrawal from 


* I am indebted to Larry Amsel of Columbia University for our discussions related to this topic. 
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society and friends, as well as hyperarousal, increased vigilance, and increased startle 
responses, which we can take as evidence of Pavlovian action-selection (Chapter 8) and 
motivational (Chapter 13) learning. These startle responses can be extremely danger¬ 
ous in well-trained soldiers, who have learned to associate violent action responses with 
those situations (Chapter 10). 

Like addiction and problem gambling, PTSD seems to be an interaction between 
inherent vulnerability and external cause. For example, it is not true that everyone 
exposed to a difficult or traumatic situation gets PTSD. 6 There is now evidence that there 
is a significant difference in hormonal levels and several brain structures between soldiers 
and other patients who end up with PTSD after an incident and those who don’t. 7 Twins 
without PTSD (who share the same genetic code) of soldiers who got PTSD reacted 
more strongly to startling cues than twins of soldiers who didn’t get PTSD from simi¬ 
lar experiences. 8 Similarly, the hippocampus (a brain structure we’ve seen to be deeply 
involved in memory, see Chapter 16) is physically smaller in twins of soldiers with PTSD 
than twins of soldiers without. 9 As with all of these vulnerabilities (like the genetic vul¬ 
nerabilities we’ve discussed elsewhere), these are correlations. People with smaller hip¬ 
pocampi may be more likely to end up with PTSD than people with larger hippocampi; 
however, there are lots of other factors involved. Simply having a smaller hippocampus 
does not mean that you will get PTSD if you are ever exposed to a trauma. 

Three theories have been proposed to explain PTSD: (l) that PTSD patients have 
stored a “flashbulb” memory, which cannot be forgotten, 10 (2) that PTSD patients 
have difficulty extinguishing fear-related memories, 11 particularly through altered fear 
processing, 12 and (3) that PTSD patients have difficulty identifying the context of mem¬ 
ories, allowing those memories to intrude on other contexts. 13 A fourth possibility is that 
PTSD patients have difficulty consolidating memories from hippocampal-dependent 
episodic representations into hippocampal-independent semantic representations 
(Chapter 16). We will address each of these theories and then try to bring them back 
together into a coherent picture of decision-making and PTSD. 


Flashbulb memories 

The first psychological theories of PTSD suggested that the traumatic event produced 
a flashbulb memory, a perfect photographic memory of an instance. The concept was 
that the stress and intensity of the memory storage overwhelmed the cortical storage 
systems, “burning it into the brain.” 14 This theory, however, predicts that the memory 
will be recalled identically each time, which is not how these memories occur. 

Although the original definition of flashbulb memories put forward by Roger Brown 
and James Kulik recognized that the flashbulb illuminated only part of the picture, the 
concept remained that the illuminated images remained steady. In fact, Jennifer Talarico 
and David Rubin found that, although they remained vivid, “flashbulb” memories 
(defined as memories that stayed for long periods of time) decreased in consistency 
over time, suggesting that they were being reconstructed each time. 13 Talarico and 
Rubin suggest that the primary difference between flashbulb memories and more com¬ 
mon, everyday memories, was the internal confidence in it, rather than in the properties 
of the recalled memory itself. 
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Interestingly, this suggests that although the traumatic memories are vivid and intru¬ 
sive, they are constructed, much like ordinary memories, and also suggests a relation¬ 
ship between episodic memory and PTSD. 


PTSD and fear associations 

We have already seen (in Chapter 4) how animals extinguishing responses to learned 
associations change their behavior because of additional components entering into the 
equation, not because they have forgotten the learned association. 16 The addition of a 
new process rather than the forgetting of the original association leads to spontaneous 
renewal of responses and cued recovery of those responses. 

Central to this concept is the recognition that there are multiple memory (and 
action-selection) systems. In particular, there is a Pavlovian (emotional) action-selection 
system that works through direct connections from sensory thalamic systems to the 
amygdala. 17 As we saw in Chapter 8, our Pavlovian responses produce visceral, emo¬ 
tional reactions to stimuli, sometimes even without our knowledge. But Pavlovian 
mechanisms also affect the motivational system (Chapter 13) as well, which influences 
the Deliberative and Procedural systems (see Chapter 6). This means that a Pavlovian, 
emotional response could label a memory particularly salient and particularly likely to 
be retrieved. 

However, we should be wary of simply saying that PTSD patients have difficulty 
extinguishing their behaviors. There are several key differences between the sort of 
associative learning that occurs in typical extinction paradigms and PTSD. Extinction 
paradigms usually entail multiple exposures to the cueing stimulus to create the original 
association, and the reaction to the association is common among all animals exposed. 18 
Spontaneous reappearance of the association occurs after long time periods, not imme¬ 
diately after the event. 19 Cued reappearance of the association occurs with a re-exposure 
to the specific cues that created the original association. 20 In contrast, PTSD generally 
occurs in response to a single, traumatic event, the intruding memories occur quickly 
after the event and fade with time, and the intruding memories appear, even uncued, in 
situations with little or no relationship to the original association. 21 


Memories out of context 

Lynn Nadel and W. Jake Jacobs have suggested that PTSD patients have specific dif¬ 
ficulties in separating contexts, which allows the memories to reappear (to intrude) 
in situations where they would normally not. 22 Specifically, they propose that gluco¬ 
corticoids in the hippocampus that arise during stress disable the hippocampus dur¬ 
ing memory storage, allowing the memory to become unlabeled spatiotemporally, thus 
allowing the memory to become, in a sense, unstuck in time. Stress affects the stor¬ 
age and retrieval of memories, primarily through the action of glucocorticoids (such 
as cortisol). 23 In particular, Nadel, Jacobs, and their colleagues have found that stress¬ 
ful events prior to an experience facilitate the storage of emotional memories through 
a hippocampal-amygdala interaction, even while they impair hippocampal processing 
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on hippocampally-dependent memory tasks. 24 Although stress during retrieval impairs 
that retrieval (it’s harder to remember things under stress), stress immediately following 
training increases storage and later retrieval (it’s easier to remember things that happen 
to you when you are stressed). 25 

This interaction (between salient, value-marked memories and uncoupled hip¬ 
pocampal memory) has been suggested as a key factor in PTSD by several researchers. 26 
In these theories, PTSD occurs when stress levels enhance amygdala learning and sup¬ 
press hippocampal learning. 27 Nadel and Jacobs suggest that the effect of the massive 
stress event is to enhance learning in the amygdala, decrease learning in the hippocam¬ 
pus, and produce a highly value-laden but contextually-uncoupled memory. 28 These 
theories, however, suggest that the PTSD patient will not be able to provide a spa- 
tiotemporal image of the event, particularly the patient’s location within the sequence 
of the event, which does not seem to fit with the descriptions of the vivid nature of the 
traumatic memory in PTSD. 29 


From episodic to semantic memory 

We saw in Chapter 16 how the normal progress of memory is a transition from episodic 
(you are there ) to semantic (I remember that) memories. An intriguing possibility is that 
PTSD occurs not when the hippocampus is unable to create an episodic memory of 
the event, but rather when the hippocampal-cortical system is unable to consolidate 
the memory into semantic storage. As we discussed in Chapter 16, although early stud¬ 
ies of consolidation suggested that memory transferred intact from hippocampally- 
dependent structures to hippocampally-independent structures over time, more recent 
studies have suggested that certain functions of memory remain hippocampally- 
dependent forever. 30 Recent studies have suggested that the transferred memories have 
a fundamentally different character than recently stored memories—recently stored 
memories (dependent on the hippocampus) are stored in an episodic manner, while the 
consolidated ones are stored in a semantic manner extrahippocampally. 

These theories are consistent with new views of the hippocampus as playing a key 
role in mental time travel, both of reconstructing the past and of imagining the future. 31 If 
normal recovery from trauma entails a consolidation of a memory away from an episodic 
(you-are-there, return-to-the-past) memory into a semantic ( narrative , detached ) memory, 
then PTSD patients may be stuck in the episodic mode. They may be unable to accom¬ 
plish this transformation and continue to reconstruct their past every time they access 
the memory, leaving the memory with an immediacy that intrudes on normal function. 

This is, in fact, consistent with the original studies on flashbulb memories, 32 which 
suggested that the normal “flashbulb” memory was stored, replayed and rehearsed, and 
decreased in its emotional impact with time. Importantly, Brown and Kulik were study¬ 
ing normal people remembering distant tragedies (e.g., Where were you when you heard 
the World Trade Center had been hit by terrorists?), not people with PTSD involved in 
the event itself (e.g., firefighters escaping from the falling buildings). Brown and Kulik 
hypothesized that eventual accounts of flashbulb memories were derived from contin¬ 
ued covert and overt rehearsal of these stories, translating them into a narrative and 
perhaps even verbal form. 
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This hypothesis rests on the concept that the hippocampus in PTSD patients is weak 
or dysfunctional, not nonfunctional. Patients with complete hippocampal lesions do 
not show any episodic memory recall events at all, and when they recall past events or 
imagine future events, their descriptions are primarily semantic, not episodic. 33 PTSD 
experiences are vivid and spatially and temporally located, with strong suggestions of 
mental time travel, all indications of episodic memories, and a dysfunctional (but not 
nonfunctional) hippocampus. 4 


Summary 

Attempted treatment for PTSD has included pharmacological treatments, attempts to 
extinguish the memory by acquiring safety cues, and attempts to recontextualize the 
memory, effectively transferring it from the emotionally-tagged episodic to the more 
detached semantic systems. 34 Although there is still a lot of work to do determining the 
vulnerabilities that engender PTSD, the mechanisms by which it creates dysfunction, 
and what might serve as potential treatments, it’s clear that the mental disorder known 
as PTSD has a physical manifestation in brain function. 

What is clear is that PTSD entails a complex and dysfunctional interaction between 
memory systems. Somehow, the emotional and episodic memory systems have become 
entangled in a way that is difficult to disentangle. Because they are reconstructed each 
time, PTSD memories are not “flashbulb memories,” yet they are, in a very real sense, 
burned into the brain. PTSD includes problems with emotional memories, which affect 
Pavlovian motivational and action-selection systems, but PTSD also includes hippocam¬ 
pal interactions, particularly with episodic memory retrieval. PTSD patients seem to 
have a dysfunctional hippocampus, unable to transition memories from episodic to 
semantic representations with time. 

Books and papers for further reading 

• Jonathan Shay (1994) Achilles in Vietnam: Combat Trauma and the Undoing of 
Character. New York: Scribner. 

• Frontline (2009). Soldier's Heart. PBS. 

• Joseph E. LeDoux (1996). The Emotional Brain. New York: Simon and Schuster. 

• Chris Cantor (2005). Evolution and Posttraumatic Stress: Disorders of Vigilance and 
Defence. London: Routledge. 


A In a similar vein, Alzheimer’s patients are well known for their ability to re-experience episodic 
memories. This suggests that both PTSD and Alzheimer’s patients have a hippocampus that is acting 
incorrectly, rather than one that is simply inactive. 
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Computational Psychiatry* 


dragonfly buzzing 
every which way 
Can’t get out! 


Current psychiatric categorizations of patients are based on a tax¬ 
onomy of symptoms. While this categorization seems to be reliable 
(in that different practitioners tend to categorize similar patients in 
similar ways), there seems to be a disconnect between the categoriza¬ 
tion and the underlying mechanisms. The detailed understanding of 
decision-making systems presented in Chapters 6-15 and the engi¬ 
neer's view on the brain looking)orfailure modes, vulnerabilities, and 
dysfunction (Chapters 18-20) provide a new view on psychiatry, 
which is now emerging as afield called "computationalpsychiatry." 


Our discussions of addiction, problem gambling, and PTSD suggest that the neurosci¬ 
entific view of the decision-making system has implications for psychiatric disorders. 
Since the introduction of the DSM-III in 1980, A psychiatry has been developing a tax¬ 
onomy of categories for identifying patients, presumably under the assumption that 
patients who fall in the same category should require similar treatment. Although these 
categories seem to be reliable in that different psychiatrists tend to categorize example 
patients into similar categories (that is, they agree on what categories look like), these 
categories have not been as good at identifying treatments for patients. 2 

In part, this arises because many of these categories are disjunctive; they are made 
up of subcomponents linked by an “or” relationship—if you have symptom A or symp¬ 
tom B then you are in category C. 3 If the two symptoms included in the disjunctive cat¬ 
egory are both reliable, then the disjunctive combination of them will also be reliable. 


* I am indebted to Larry Amsel of Columbia University for our discussions on this topic. 

A The Diagnostic and Statistical Manual of Mental Disorders is the go-to guide for identifying psychi¬ 
atric illness. The current edition is the DSM-IV-TR (“TR” stands for “text revision”), and the DSM-V is 
in the works. The other major manual used by clinicians for the identification of psychiatric disorders 
is the International Classification of Diseases, known as the ICD-10. 1 
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This disjunctive categorization can be seen by the predominance of DSM categories 
based on adding up symptoms. In the DSM-IV-TR, the diagnosis of problem gambling 
can be made if the patient has five or more of ten criteria. This means that two peo¬ 
ple can both be identified as problem gamblers even though they have no symptoms 
in common. The fact that the categories are reliable does not imply that they are valid, 
that they say anything about shared mechanism. 4 

The other problem with the categories identified in the DSM is that they are based on 
symptoms. As everyone knows, a given symptom could be caused by any of a number 
of potential underlying problems. Imagine a patient who arrives at a hospital with chest 
pain. Imagine if the doctor tells the patient, “Some proportion of chest-pain patients 
require open-heart surgery and some proportion of chest-pain patients respond well to 
an antacid, so we’ll try both.” Treatment would obviously succeed better if we reliably dif¬ 
ferentiate patients who were having heart attacks from patients who had indigestion.® 

Sometimes the only option available is to cure the symptom. For a doctor looking 
down at a patient who has just lost a leg from a landmine explosion, there is nothing 
to do at the time but treat the wound and eventually provide a mechanical, replace¬ 
ment leg. Similarly, the inability to initiate movement (akinesia) and the slowness of 
movement initiation (bradykinesia) seen in Parkinson’s disease are due to the loss of 
dopamine cells in the brain. 6 Just as it would be great to address the underlying causes 
that placed the landmine in the field 7 (Ain’tgonna study war no more), it would be great 
to prevent the loss of dopamine neurons in the first place. However, the only treatment 
option for a patient presenting with Parkinson’s disease is treating the symptoms, either 
by increasing the dopamine available to the surviving cells (such as through dopamine 
precursors like levodopa or through attempts to replace the lost cells) or by allowing the 
brain to sidestep whatever it was that the dopamine cells were providing (as is done in 
deep-brain stimulation). 8 

Similarly, sometimes the right treatment is to prevent the symptoms from doing 
too much damage to the body and allowing the body to heal itself. It is the symptoms 
caused by cholera that kill, not the disease itself. Rehydration is a reliable treatment that 
effectively cures the disease by treating the symptoms. 9 

At this point, there is a recognition that there is a problem in psychiatry, but practi¬ 
tioners are still debating how to restructure the taxonomy so as to get at the mechanisms 
we do know, while allowing for the fact that there are a lot of mechanisms we don’t 
know, all while continuing to do our best to treat patients, many of whom desperately 
need help. 10 A new field of computational psychiatry seems to be emerging to make these 
connections. 11 The term is meant to invoke the use of theoretical constructs (like those 
we’ve been exploring in this book) to redefine the categories and proposed treatment 
for psychiatric disorders. 

The question at hand is thus What are the mechanisms and causes that underlie psychiat¬ 
ric illness? What is depression? What drives it? What is schizophrenia? How do we define 
it? What is an anxiety disorder? Psychiatry has identified lots of different psychiatric 


B In actuality, it can be difficult to differentiate the chest pain of a heart attack from the chest pain of 
indigestion without an EKG, 5 which is why one should not try to diagnose oneself at home. Of course, 
this just underscores the issue of understanding the causes and mechanisms, rather than just the symp¬ 
toms, and having clinical tests that can reliably differentiate those causes. 
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illnesses: the DSM-IV-TR has over a thousand different diagnoses. How many of these 
are the same dysfunction manifesting itself in different ways? Is “social phobia” (fear of 
being out in society) the same or a different disorder than “agoraphobia” (fear of places 
where escape or help might be difficult) ? How many of these are different dysfunctions 
manifesting as a single symptom? (See our discussion of addiction in Chapter 18.) 

Anxiety disorders, for example, entail a pathological fear of taking actions in spe¬ 
cific situations. In humans, anxiety manifests itself in terms of fear and worry about 
future outcomes. In animals, anxiety manifests itself in observable avoidance and fear 
behaviors. 12 The circuits and brain structures involved are similar in both animal and 
human anxiety experiments. 13 They involve the structures we’ve examined as part of the 
Pavlovian action-selection system (Chapter 8) and structures we’ve examined as part of 
the Deliberative action-selection system (Chapter 9). Current models suggest that there 
are two components that drive anxiety. First, direct negative expectations of outcomes 
can drive fear-related actions through the periaqueductal gray (PAG) and the amygdala, 
which we saw earlier to be a key part of the Pavlovian fear circuitry ( the rustle in the grass 
that signals the lion; the tone that came before the shock). Second, indirect negative expecta¬ 
tions of outcomes can be generated by hippocampal-prefrontal interactions that are then 
evaluated as likely leading to a negative outcome 14 (don’t go down that dark alley!). 

In fact, when rats or mice come to an open space, they tend to hesitate, leaning out and 
then returning back to shelter, as if they are nervous about whether to go or not. 15 This 
“stretch-attend posture” is very similar to the vicarious trial and error behavior that we saw 
in Chapter 6. Like vicarious trial and error behavior, the stretch-attend posture behavior 
depends on the hippocampus, the prefrontal cortex, and their interaction. 16 This suggests 
that this hesitation likely entails predictions of future dangers. Of course, rats and mice are 
prey animals and dangers really do lurk around every corner. Humans are generally at the 
top of the food chain and tend not to worry as much about being eaten by cats or owls. c 
Generally humans are more concerned with social dangers, but the same stress-response 
mechanisms seem to be involved. 18 Anxiety attacks can come from a variety of sources, 
including social interactions (such as public speaking), financial futures, personal danger, 
danger to one’s friends or family, availability of escape and rescue (in claustrophobia and 
agoraphobia), and even rare circumstances (such as earthquakes or buildings collapsing). 
The DSM-IV-TR considers PTSD a form of anxiety disorder; certainly, the description of 
soldier’s heart sounds like a panic attack. 19 As another example, anxiety is now considered 
to play an important role in obsessive-compulsive disorder (OCD). Interestingly, OCD 
seems to be a combination of an anxiety disorder about future dangers combined with an 
uncertainty about the memory of completed actions. 20 

Computational models are beginning to be built to explain fear and anxiety. While 
they haven’t yet led to changes in treatment, there is consensus building as to the underly¬ 
ing mechanisms. We’ve already discussed addiction, gambling, and PTSD in the previous 
chapters. Similarly, other psychiatric diseases are beginning to be addressed by compu¬ 
tational models—for example, depression, schizophrenia, and other disorders. 21 

There are a number of major psychiatric disorders that are related not to individual 
action-selection systems, but rather to social interaction systems—for example, border¬ 
line personality disorder and sociopathy or psychopathy. 22 These interactive dysfunctions 


Except perhaps in Twin Peaks, where the owls are not what they seem. 17 
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entail people who do not show normal social interactions. We will leave these interactive 
issues aside until we can address neuroeconomics, game theory, the propensity for humans 
to organize into groups, and the mechanisms through which cooperation within those 
groups is maintained (in Chapters 22 and 23). 

With all of these examples (as with all psychiatric disorders), there is an impor¬ 
tant issue that it is extremely difficult to define the line between normality and dys¬ 
function. 23 There has been a lot of discussion in recent years about the overdiagnosis 
of attention-deficit/hyperactive disorder (ADHD). 24 Where is the line between a 
rambunctious boy who is not getting time to run every day and ADHD? How much 
is ADHD being used as a diagnosis to control overcrowded classrooms? Similarly, is it 
really true that one American in ten is depressed? (in 2008,10% of Americans were tak¬ 
ing antidepression medication, presumably mostly selective serotonin reuptake inhibi¬ 
tors [SSRIs]. 25 ) How often are these SSRIs being used to increase confidence, and how 
much are they being used to treat normal grief? 26 

On the other hand, it is important to recognize that there are people with severe psy¬ 
chiatric dysfunction who desperately do need help. True depression and normal grief are 
very different things. 27 Suicide is the tenth leading cause of death in the United States. 28 
Similarly, while some rambunctious children may be incorrectly diagnosed with ADHD 
and treated with Ritalin, there are others who really do have a problem and for whom 
Ritalin is the difference between getting an education or not. Trichotillomania (hair pull¬ 
ing) is not about the 10-year-old who twirls her hair nervously while shuffling her feet 
and smiling cutely up at you, nor is it about pulling that first gray hair. Trichotillomania 
is about people who have pulled their head bald and their scalp bloody. 29 There are 
people with severe dysfunction who do need help. 

At this point, we don’t have clear answers. Although this is not a new problem and 
has been recognized as an issue within psychiatry for many years, 30 there does seem 
to be a change coming where the connections are being identified between neurosci¬ 
entific mechanisms and psychiatric dysfunction. 31 What these mechanisms are, where 
the severe dysfunctions are, and how to treat them are all open questions being actively 
explored within the neuroscience, psychological, and psychiatric communities. 
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What Makes Us Human? 


This stadium once held twenty thousand, 
but the river moved, and now only 
tourists stand on its crumbling walls. 


Humans are animals with unique abilities, particularly in the 
realms of language, imagination, and culture. However, like our 
physical bodies, our mental abilities contain echoes of our related 
mammalian cousins. Like our decision-making systems, our tre¬ 
mendous abilities have precursors in other animal behaviors. 


I am fascinated by history and love to travel and see the old ruins of great civilizations. 
I have stood on the walls of Troy and looked out over the plains of Hisarlik where 
Achilles and Odysseus once stood. I have stood on the stage of the ancient Greek city 
of Miletus and sat in the stands of the Roman stadium of Ephesus. I am always fasci¬ 
nated by how much remains constant. The defensive fortifications of a medieval castle 
in Japan (such as Himeji Castle) are similar to those of a medieval European fortress 
(such as Harlech Castle, in Wales). The components of an ancient Roman stadium are 
instantly recognizable to anyone who has attended a modern sports stadium, the rows of 
benches (numbered and labeled by row), even the concession stands. Similarly, we can 
appreciate narratives from the world over. The modern reader recognizes Achilles’ petu¬ 
lance at Agamemnon’s demands in Homer’s Iliad and Odysseus’ war-driven exhaustion 
in the Odyssey. The trickster Anansi continues to capture children’s imaginations, 1 and 
the adventures of dArtagnan and the Three Musketeers 2 still spark the interest of young 
boys and girls everywhere. Whether it be the graffiti on the wall of the boys’ dormitory in 
Priene (a ruined city in Turkey) or the shape of the council seats in Wupatki (in northern 
Arizona), we recognize familiar human interactions in our ancient ancestors. Something 
is constant through the ages, something that makes it possible for us to look at a set of 
drawings on the cave walls of Lascaux and know that some person painted them. 

Certainly, there is a qualitative difference between humans and the other animals 
that have been mentioned in this book. Beavers may create dams of wood that mod¬ 
ify their local environment, but they have never put a man (a beaver?) on the moon. 
Outside of literature, no rats have ever built cities or written novels. 3 But as we have 
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seen throughout this book, our decision-making system, much like our physical bodies, 
contain echoes of our related mammalian cousins. If we evolved from animals, then how 
are we different from them? Even the phrase “evolved from animals” implies a trajectory 
from animals to humans, with humans placed above and beyond. A better statement 
would be that we are part of the evolutionary history, a moment in evolutionary time. 
We are not evolved from animals—we too are animals. 

There is much about humans that seems unique among animals. No other animal 
has language skills like ours. (Although it has been possible to teach some animals the 
rudiments of language, none of them has ever achieved language skills beyond that of 
a young child. 4 ) The complexity of the cities we build and our individual specializa¬ 
tions within those cities are unparalleled. (Although termites and other social insects 
also build large hives, these hives are made of a limited number of specific individual 
castes and do not resemble the flexible complexity of human cities. 5 ) And we seem to 
be destroying our ecosystem at an unprecedented speed. 6 However, as is nicely pointed 
out by Matt Ridley in The Agile Gene, lots of species are unique. Termites build in mud, 
squids squirt ink, and cobras spit venom. Simply being unique is not enough. The 
question is What makes us unique? 

Jane Goodall, who has studied a tribe of chimpanzees in Gombe for the past 50 years, 
complained that every time she found her chimpanzees showing signs of what makes us 
human, the philosophers would move the goalposts. 7 At first, humans were tool users. 
But as early as the 1920s, the psychologist Wolfgang Kohler found that apes could 
learn to stack boxes purposefully to find reward. 8 In the field, Jane Goodall observed 
chimpanzees using twigs to pull ants and termites from tree stumps. 9 

Then humans were the only ones to make tools. But then Goodall observed that the 
chimpanzees were actually making their tools, finding the right stick, cleaning it off, and 
preparing it. 10 (it is now known that otters and crows and many other species also use 
tools. Otters will use rocks to break open sea urchins, and crows have been found to 
make tools out of twigs and leaves to fish for insects. 11 ) 

Then it was suggested that humans are the only ones to have developed separate cul¬ 
tures. But chimpanzee tribes use different tools to attain food. In the Tai Forest in the 
Ivory Coast, a chimpanzee tribe uses rocks to crack kola nuts (which have a very hard 
shell). Chimpanzees from the Gombe tribe do not know how to use rocks to crack kola 
nuts, and chimpanzees from the Tai Forest tribe do not know how to fish for termites. 12 
These different abilities are spread from chimpanzee to chimpanzee culturally. 

Cultural abilities have even been observed as they were discovered. In the 1960s, 
Japanese macaque monkeys began to wash sweet potatoes in the ocean to remove dirt 
and to add salt. 13 The ability spread first from the discoverer (a young female) to others 
she knew, and then to the others in the tribe. It followed trails of kin, parenting, and 
training. Subsequent observations of cultural behaviors have found that they generally 
spread through celebrity and dominance such that higher-rankprimates (who are closely 
observed by lower-rank ones) spread the meme faster than lower-rank primates. 14 

So other animals show tool use, and other primates have the presence of culture 
(or at least a precursor to culture). What makes humans different? 

One suggestion has been that humans are the only ones to think about other places 
and other times, to (as it were) “mentally time-travel.” 15 But, as we’ve seen, rats show 
search behaviors when making choices and seem to represent expectations of the 
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outcomes of their actions. 16 Nikki Clayton at Cambridge has seen that scrub jays recog¬ 
nize when and where they cached food, can take into account how long that food will 
remain edible, and can take into account the likelihood that another jay is aware of the 
hiding location of the food cache. 17 Both monkeys and rats can represent alternative 
options that might have been when they make errors (as in “I could have got reward if 
I had made the other choice”). 18 

That being said, it is not clear that scrub jays can maintain episodic memory beyond 
food caches, 19 that monkeys or rats can imagine far-distant futures orpasts, or that chim¬ 
panzee culture extends to the kind of narrative stories that we tell ourselves. I suspect 
that the difference is going to be one of quantitative differences becoming a qualitative 
one. Nevertheless, the things that make us human have their roots in our evolutionary 
history. 20 

Careful observations have shown that other animals play, laugh, fight, even mount 
what, for lack of any other terminology, can only be called wars. 21 And yet, there are 
differences. We have a flexibility in our decision-making systems that is far beyond what 
other animals can do. We communicate with language that is infinitely flexible and capa¬ 
ble of depth of meaning. We can work together in groups larger and more complex than 
other species. In particular, our groups come and go. We are capable of shifting groups 
with context more easily than other species. 22 As Aristotle defined us, we are the “politi¬ 
cal” animal. 23 

Our social identity can be remarkably dynamic and flexible. We have extended 
families, tribes, and nations, but also groups based on the places where we work, our 
churches, mosques, and temples, and our sports teams and political parties. 11 People 
who work together as part of the same group in one context (at a job or cheering for a 
sports team) can dissolve into anger and intractable argument in another context (such 
as politics). 

Some philosophers attribute the difference to language, 25 which has several impor¬ 
tant properties that may explain our differences. First, language can communicate very 
detailed information (such as plans or instructions) quickly. 26 Second, it can store 
knowledge outside of an individual (as in books or webpages). 27 Third, language pro¬ 
vides an interactive social network and marks our place within it. 28 

We often use language as actions in themselves. 29 A person saying “I do” at his or 
her wedding is committing to a specific contractual deal. As pointed out by Orson 
Scott Card in his novel Speaker for the Dead, a marriage is a contract between three 
parties—the two people being married and the community that agrees to treat them 
as married. 30 

Language can also be used to build on itself—one person can provide part of the 
answer, and another can build on it. 31 This is the misnamed “wisdom” of the group, 
which is not so much wisdom (groups can be very unwise) but rather a positive feed¬ 
back process of knowledge. 32 One of the major breakthroughs that led to the scien¬ 
tific revolution and our modern world was the development of the scientific journal 


A In one of the strangest disasters I have witnessed, politics has become a sports question of who is 
winning rather than a policy question of what is the right thing to do. This has led to political tribalism, 
in which the leader of a party can say publicly that he is more interested in the other party’s leader los¬ 
ing the presidency than in fixing the problems that are facing his country. 24 
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in the 1600s. B A scientific article reports a single result or finding, and, although it 
is generally a complete story, it does not have to completely answer all the questions 
that might arise. Before the journals, science was reported in books and monographs, 
which required decades to write. This meant that other scientists could not follow up 
on ones work until it was complete. By providing a means of reporting a partial answer, 
scientific journals allowed scientists to point other scientists in the right direction, to 
put a single brick in the edifice of science rather than having to construct an entire 
building by oneself. 

The other thing that language can do is point. Language allows us to coordinate our 
goals and to coordinate our attention. 33 It can point a direction, identify a danger, a tar¬ 
get, or an outcome. The level to which other animals can track referents is still being 
debated, but they do not seem to track them as easily as humans do. For example, 
humans are the only ape with white sclera, which enables others to know where we are 
looking. 34 Much of our social interaction is based on knowing where the other person is 
looking. Take, for example, the importance of eye contact in conversation. 35 (Of course, 
knowing where someone is looking is not always a good thing, as any intercepted NFL 
quarterback knows.) 

Fundamentally, however, one of the largest differences between humans and other 
animals is in what has traditionally been called morality . i6 Jane Goodall observed sociop- 
athy in her chimpanzees—a mother and daughter (named Passion and Pom) developed 
a cannibalistic procedure attacking new mothers and their infants where one of them 
would distract a new mother and the other would attack and kill the infant. 37 If one of 
the large males was in the area, he would defend the new mother and infant, but the 
males did not attack Passion and Pom outside of the context of their crime. Humans do 
not tolerate such sociopaths. 38 

Whether the difference is a qualitative leap (a change in kind that can be clearly sepa¬ 
rated) or merely the repeated effect of small quantitative changes (changes in level or 
depth) is still unknown. Yet, something is different. Termites may build mounds, and 
other animals may build nests of incredible complexity, but those mounds are not cit¬ 
ies on the scale of New York or Tokyo or Alexandria. Chimpanzees may fight wars, but 
those wars do not include machine guns, tanks, or nuclear bombs. c Bower birds may 
create art, but that art is not disseminated to and discussed by millions of bower birds 
all over the world. Baboons may live in groups, but those groups do not “friend” each 
other on Facebook. 

There have been fascinating discussions of just how intelligent some species are. 
Elephants have highly complex societies, calculate expectations for the availability of 
fruit and water as a function of recent weather patterns, and remember their dead for 
years. 39 Vervet monkeys lie about attacking predators to escape bullies. 40 And whales 


B The first scientific journal was the Philosophical Transactions of the Royal Society, which has been pub¬ 
lishing scientific papers since 1665, including the first scientific reports from Newton, Darwin, and others. 

c Whether this is a good thing or not is, of course, debatable. As pointed out by Douglas Adams in 
his classic The Hitchhiker's Guide to the Galaxy, “man had always assumed that he was more intelligent 
than dolphins because he had achieved so much—the wheel, New York, wars and so on—whilst all the 
dolphins had ever done was muck about in the water having a good time. But conversely, the dolphins 
had always believed that they were far more intelligent than man—for precisely the same reasons.” 
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and dolphins sing amazingly complex songs that depend on their cultural and societal 
interactions. 41 Although something may be different about humans, the things that fun¬ 
damentally make us human, our language, our minds, and our decision-making systems, 
like our physical bodies, share commonalities with and derive from our animal nature. 

In the next two chapters, we will explore two of those uniquenesses: morality and 
consciousness. 
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The Science ofMorality 


Fear grips a nation. 

The destruction of barbarians at the gates 
are no match for the destruction 
of the barbarians within. 


In much of current discussion, morality is said to be outside of the pur¬ 
view of science because science is descriptive, not prescriptive. Simply 
said, this means that science cannot tell you what to do. Although 
there is certainly a clear and important difference between description 
and prescription, the implications of the science of decision-making 
for morality are more complex than they would seem at first glance. 


Description and prescription 

Everyone agrees that there is a fundamental philosophical difference between descrip¬ 
tion and prescription. In the famous words of David Hume, there is a difference between 
“is” and “ought to be .” 1 Description is about what will happen, (if you drop that heavy 
rock, it will fall. If you drop it on your foot, it will hurt.) Prescription is about what you 
should do. (Don’t drop that rock on your foot!) 

This distinction is very clear in modern medicine. Description is the effect of the 
treatment. (Antibiotics are ineffective against viruses. Bacteria evolve in response to 
antibiotics.) Prescription is what we should do to treat the disease, (if you have a bacte¬ 
rial infection, antibiotics are a good idea. If you have a viral infection, don’t use antibiot¬ 
ics because you are only helping the bacteria in the world evolve resistance to them.) 

Science is fundamentally descriptive. Science is a method to determine the causal 
structure of the world through prediction and replication/ We need a different term 


A The complex issues of the sociology of science and whether science works by explicit falsification 
or through integrated framework construction 2 are not important for the debate here. For those inter¬ 
ested, there are lots of books on this issue. My favorite is Just a Theory by Moti Ben-Ari. In any case, let 
us simply say here that, over the long term, our control of the world has improved because scientific 
theories have become more accurate descriptions of reality over the centuries. 
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for how to make prescriptive decisions between individuals and groups who disagree. 
These ar e policy questions, and the best term, I think, is politics . 5 Saying that prescriptive 
decisions are fundamentally political questions is not meant to imply that they depend 
on one’s party allegiance (capital-P Politics) or the specifics of what gets written into 
laws (morality is not legality)—we are talking small-p politics here: How do humans 
interact with other humans and their societies? As we will see below, humans spontane¬ 
ously organize into small-p political structures to handle their social interactions, and 
much of that small-p politics is unwritten social constructions . 3 

As we will see, the role of small-p politics becomes clear when we disagree about 
goals. If one kid wants to play basketball with dad outside and the other kid wants to 
play a computer game with dad inside, some negotiation has to occur to settle a fair dis¬ 
tribution of dad’s time. Similarly, a company may decry regulations that are preventing 
it from opening new stores and new markets, while others demand those regulations to 
ensure a locally desired principle. Or one may need to make a decision about whether 
to risk prospecting for oil in fragile shrimp-farming grounds. Notice that I have been 
very careful in these examples not to make any claims about right and wrong; I have 
only shown examples where goals differ and negotiation is required. Right and wrong in 
each of these examples will likely depend on the details of the situation. For example, it 
makes a big difference if those regulations are preventing the company from dumping 
toxic waste into a neighboring city’s drinking water or if those regulations state that only 
white people are allowed to own companies. 

The statement that goals differ does not imply that both sides are morally equivalent 
(the members of one side may say that their goal is to exterminate all people of a certain 
ethnicity, while the other side may say that they want to both live in peace). Similarly, the 
statement that negotiation is required does not mean that an accommodation has to be 
made—in the American Civil War and World War II, the “negotiation” between two 
opposing moral codes (slavery or not, fascism and genocide or not) entailed the two 
sides fighting a war. 

There are three literatures that have tried to claim control of the prescriptive magis- 
teria of morality: religion, philosophy, and economics. 

Religion 

In the separation of “is” from “ought to be,” morality is often seen as the purview 
of religion and philosophy . 4 Stephen Jay Gould famously argued for what he called 
“NOMA,” or non-overlapping magisteria, and tried to define the difference between 
science and religion as being of different “magisteria” (different realms of thought) — 
religion could remain about what we should do, while science could remain about 
what happens when we do something. Religion could be about meaning, while sci¬ 
ence could be about the physical world. Religion could be about why, while science 
could be about how. 


B I am indebted to Peter Collins of Salford University, whose presentation at a small conference 
organized by Don Ross in 2007 clarified this distinction with the simple, startling sentence: “Science is 
about what happens when we do something. Politics is about what we should do, given that we disagree 
about our goals.” 
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Although many people have argued that morality is the purview of religion, I will 
argue here that this is incorrect . 0 In fact, religion is also descriptive, not prescriptive. The 
prescription of religion is based on creating a descriptive theory in which the prescrip¬ 
tion becomes obvious—God says this is what you must do. If you do this, you will be 
rewarded. The reward is usually some sort of heaven, while punishment is hell . 6 (Even 
religions without gods describe reward and punishment for correct behavior, such as 
the ideas that appropriate behavior will allow one to reach nirvana, or that good deeds 
will lead to reincarnation into a better life in the next cycle.) But reward can also be sug¬ 
gested to occur in this life, such as the prosperity theology movement arising recently 
in the United States. 

Notice that this is fundamentally a descriptive statement: If you do the right thing, 
then you will be rewarded. The fact that this is descriptive can be seen in Pascal’s famous 
wager in which he has to decide whether or not to believe in God, and decides to do 
so because the cost of belief if God does not exist is small, while the cost of disbelief if 
God does exist is large . 7 The bet is laid down on the accuracy of the description. The 
prescription comes from the goal of attaining reward (Heaven) and avoiding punish¬ 
ment (Hell). 

The separation of prescription from religion can be seen in George Bernard Shaw’s 
play Man and Superman, where several of the characters reject the goal (rejecting pre¬ 
scription but not description) and decide they would rather go to Hell than suffer in 
Heaven. Similarly, in Milton’s Paradise Lost, Lucifer famously says, “It is better to rule 
in Hell than to serve in Heaven.” We can say that these people are misguided, that they 
have made the wrong choices (in Milton that’s clear, in Shaw not so much), but the fact 
that a change in goals changes the morality implied by religious belief makes clear the 
inadequacy of religion as the prescriptive magisteria. 

Some nonsupernatural moral codes are often referred to under the rubric of religion, 
such as Confucianism, Taoism, and Utilitarianism . 8 However, these are really examples 
of optimization functions—Confucianism’s stated goal is to maximize stability within 
a society, while Taoism aims for “inner peace” and harmony with one’s environment, 
and Utilitarianism attempts to maximize total happiness. These are prescriptive state¬ 
ments—they define a goal to optimize and say that we “ought” to do things that maxi¬ 
mize that goal. The discussion of optimization functions leads us first into philosophy 
and then to economics. 


Philosophy 

Historically, philosophers have been searching for a categorical imperative, a simple 
statement that can be used to define morality from first principles . 9 The classic exam¬ 
ple is of course the Golden Rule, “Do unto others as you would have them do unto 
you,” with versions in the Torah, the New Testament, and the Analects of Confucius . 10 


c These statements are not meant to imply that religion is not a useful human construct. Religion 
could serve many useful purposes, including as comfort, or as a means of organizing human groups for 
the purposes of enforcing within-group altruism. Religion may also serve as a psychological construct 
to enforce morality and encourage self-control, but the content of that morality and the goals of that 
self-control derive from nonreligious mechanisms. 5 
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The complexity of this is interesting in that versions differ in terms of whether phrased 
in positive (Do unto others what you would have them do unto you) or negative (Do not do 
unto others what you would not have them do unto you) terms. 

Many such categorical imperatives have been proposed, but, as pointed out by Patricia 
Churchland in her book BrainTrust, it has always been possible to find extreme situations 
that undercut these imperatives. Applying the rule to real situations demands exceptions, 
and we end up with a hodgepodge of rules and regulations. For example, the statement 
that we should increase the total happiness in the world suggests that one would have to 
sacrifice food for ones children to feed starving children elsewhere. While we might cel¬ 
ebrate someone who works tirelessly for starving children, there aren’t many people who 
would be willing to let their own child starve to death for another’s, even for two others. 
Similarly, the statement that we should do unto others what we would like to have done 
to us if we were that other person leads to problematic situations in the face of fanatical 
prejudices. Imagine someone who says they would rather die than live as a paraplegic. 
Obviously, we would not want that person to carry out their imagined wishes on some¬ 
one else who is paralyzed. In fact, paraplegic (and even tetraplegic) patients who become 
paralyzed from an accident tend to return to the same happiness levels they had before 
their accident . 11 (Happiness readjusts allostatically to the new situation.) 

Again, we come back to the political question of How do we negotiate different goals? 
John Rawls put forward an elegant proposal based on his theory of the ignorance of origi¬ 
nal position . 11 Rawls suggests that laws and justice should be defined in such a way that 
these are the laws one would like to see given that one does not know where one will be 
in the society before one writes the laws. So, you would not want to make a law endors¬ 
ing slavery because you would not know beforehand whether you were going to be born 
a slave or free. Rawls’ proposal is a philosophical description of a political solution. He 
explicitly states that his goal is that he starts from the assumption of a constitutional 
democracy, from the assumption that all men (all people) are created equally. 

The problem with this theory is, of course, that people are not always willing to 
assume that they could have started anywhere in society. Aristocracies, religious selec- 
tionism, and differences in individual talent can lead to an unwillingness to accept 
Rawls’ premise. People are not equally talented. Some people are smarter than others. 
Some people are stronger. Some people are more deliberative and others more emo¬ 
tional. Although these differences are influenced by genetics, they are clearly an interac¬ 
tion between those genetics and societal influences . 13 

As pointed out by Stephenjay Gould in AMismeasure of Man, these genetic effects are 
notoriously difficult to measure accurately, are dangerously susceptible to cultural influ¬ 
ences, and tend to get swamped by variability in those cultural influences. Nevertheless, 
it is easy for a successful individual (who is usually the one writing the law) to look at an 
unsuccessful individual and say that the reason for the difference in success is a moral fail¬ 
ing. Rawls’ proposal is fundamentally dependent on empathy, the ability to put oneself in 
another’s place (a critical component of both religious and philosophical moral codes 14 ). 

It can be difficult to say, “There but for the grace of God, go I.” Amoral individuals 
have explicit difficulties with empathy . 15 One of the things we will see later, when we 
directly examine what people do, is that people tend to be less willing to share if they 
think they’ve worked for their reward . 16 Even if we believe in Rawls’ premise, we will still 
need to understand how to negotiate with individuals who do not. 
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Economics—The hazards of optimization functions 

One field that has, in fact, made “scientific” claims about morality is economics. These 
claims have been implicit rather than explicit, but by our separation between descriptive 
and prescriptive, a remarkable number of economic results are prescriptive rather than 
descriptive. This is because they are defined under an optimization function. 

The most common optimization function used in economics is that of efficiency, 
which is defined as an economy with no missed opportunities. 17 Imagine that country A 
can grow carrots for $10 a bushel and celery for $20 a bushel, while it costs country B 
$20 to grow a bushel of carrots but only $10 a bushel to grow celery. This means that 
if each country grows all of its own vegetables, the two countries are wasting money— 
they are inefficient. It would be more efficient for country A to grow only carrots and 
country B to grow only celery and for country A to trade some of its carrots to coun¬ 
try B for some of the celery grown in country B. This would be more efficient even if 
there was a small cost to transporting carrots and celery across the border. If transporta¬ 
tion cost $1 a bushel, then country A could effectively grow carrots for $10 and celery 
for $11. Similarly, country B could effectively grow celery for $10 and carrots for $11. 
Economists usually discuss this in terms of monetary costs, but we get the same results 
if we think of this in terms of person-power, how many hours it takes how many people 
to accomplish something. Country A grows carrots with less work than country B, while 
country B grows celery with less work than country A. Together, they can grow more 
food with less work. This is the advantage of specialization. 

This all sounds well and good, but it assumes that the goal of an economy is to mini¬ 
mize the “missed opportunities.” The effect of this free-trade agreement between coun¬ 
tries A and B is that the celery growers in country A are now out of work, as are the 
carrot growers in country B. If your goal is to maximize jobs, then this deal is a bad 
one. The problem is that economists use words laden with morality like efficiency or 
rationality. 1S ' D An inefficient car that gets poor gas mileage is obviously a bad thing. Is an 
inefficient economy also a bad thing? 

The assumption that efficiency is the only goal is a strange one. 20;E Sometimes inef¬ 
ficiency is an obvious problem, such as when the costs and purchasing prices are too 
far out of balance and the economy grinds to a halt, or with corruption that pulls eco¬ 
nomic production out of an economy; 22 but, as with a car that gets poor gas mileage, 
efficiency is really only one element of a more complex optimization function. 23 Lots of 


D We’ve discussed the dangerous use of the terminology of “rationality” and “irrationality” through¬ 
out this book, in particular the idea that there is a rational self trying to control an irrational other. 
Many philosophers and psychiatrists and even many cognitive scientists still share this view. 19 (See the 
discussion in Chapters 8 and IS.) As I hope I’ve made clear, I am uncomfortable with this rejection of 
the nonconscious decision-making systems as irrational. 

E Why have economists latched onto “efficiency” as the goal? Some have argued that it is because 
this creates mathematically solvable problems. I find this conclusion strange because it would seem 
to me that one could just as easily define other optimization functions. Some have argued that this is 
due to an inherent feedback loop of students being taught early that efficiency is the only optimization 
function that matters. I wonder how much it depends on the self-interest of certain individuals, as this 
efficiency argument creates winners and losers in an economy. 21 
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people buy inefficient cars because they like the safety features or the ability to carry 
large objects (like lots of kids) or for their style and their look. 

In fact, these economic descriptions of maximizing efficiency are poor descriptors 
of what humans actually do, particularly at the microeconomic (individual) level, and 
require extensive additional components to properly describe behavior. 24 

At a microeconomic level, these questions of interpersonal interactions can be cap¬ 
tured in the context of game theory, 15 which assumes that two people are playing a game. 
This game is assumed to be specified as a sequence of choices (actions and their con¬ 
sequences) leading to a final set of payoffs. Each situation is a node in a tree, and each 
action takes the two players from one node to another. (Notice how this concept of situ¬ 
ation and action mirrors the decision-making definitions we’ve been using throughout 
this book.) A classic example would be the game tic-tac-toe (Figure 23.1). At the start, 
the situation has an empty board. Player X has nine options. When player X puts an X 
in one of the squares, player O now has eight options. At the end of the game, there are 
three possible payoffs: Player X wins and O loses, Player O wins and X loses, or there is 
a tie. (it can be shown that if both players play optimally, tic-tac-toe will always end in a 
tie.) Strategy is defined as the set of choices that a player would make at each point in the 
sequence. (What should Player X do at the start of the game? [The center is optimal.] 
What should Player O do when Player X has two in a row? [Block it.]) 

If neither player can improve his or her strategy by changing something, then the 
players are in a Nash equilibrium. 16 '* The problem with this is that this breaks down in 
cooperative games. (Nash’s proof of optimality was technically only for zero-sum non- 
cooperative games. 28 ) As we will see later, in cooperative games (such as the prisoner’s 




OOP 

x| |x 

O wins 


Figure 23.1 Game Theory and tic-tac-toe. Each configuration of the tic-tac-toe board 
is a state. The action of each player changes the state. Tic-tac-toe is a zero-sum game: if 
player X wins, then player O loses, and vice versa. Compare the prisoner’s dilemma game 
(Figures 23.2 and 23.3). 


F The Nash equilibrium is named after John Nash, the famous mathematician who discovered this 
as part of his Ph.D. thesis in 1950. Both Nash’s mathematics and his trials with schizophrenia were 
described eloquently in the book and movie A Beautiful Mind. 17 
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dilemma, below), a player can improve his or her gain by not cooperating, but if both 
players don’t cooperate, then they both do worse. In fact, as we will see below, people are 
remarkably cooperative, even when it is explicitly stated that games will only be played 
once and when no one knows who you are, so that there is no opportunity for reciproc¬ 
ity to affect decision-making. 29 


So what do humans really do? 

The rest of this chapter will ask the question What do humans do in their interactions? 
Rather than address the prescriptive question, I’m going to spend the rest of this chapter 
addressing what we know about human behavior (descriptively). Just as knowing the cor¬ 
rect description improves prescription in medicine (see the antibiotic example earlier), 
we can hope that a better description of human behavior may help us know what prescrip¬ 
tions are possible. 0 Just as we sidestepped the question of whether taking drugs is right or 
wrong in Chapter 18, and instead asked Why do people take drugs? and How do we change 
that (if we want to)?, I am going to sidestep the question of what is moral and instead ask 
What do people do when faced with moral (social, personal interaction) questions? 

Fundamentally, the things that we call “morality” are about our interactions with 
other people (and other things, such as our environment). Our daily interactions rest 
on an underlying societal trust (civilization). 32 As noted in the previous chapter (where 
we discussed what makes us human), these groups spontaneously organize, whether it 
be for mutual protection, for societal stability, or because well-organized groups out- 
compete poorly organized groups. Simply stated, if two armies face each other and one 
works together, but the other is made up of individually selfish members, it is pretty 
clear which group will most likely win the battle. 33 

So what does that mean for morality? How can science address morality? What does 
it mean to study morality? We know humans can be gloriously moral (Martin Luther 
King, the Freedom Riders, soldiers sacrificing their lives for their country, firefighters 
running into a burning building, donations to charity). We know humans can be glori¬ 
ously immoral (Nazis, the KKK, genocide in Rwanda, Serbia, and Darfur). The first 
question we will need to ask is What do humans do when faced with a moral situation? 
What do humans do, and what are the parameters that drive their action choices? If we 
understand how humans think about morality, then we will be able to decide whether 
we want to change it, and if so, how to change it. 


Early studies: The identification of evil 

Fundamentally; morality is about our interaction with each other. As with the other 
questions we’ve been addressing in this book; in order to study morality; we need a 


G It is very important not to fall into the sociobiology flaw. 30 The fact that something is descriptively 
true should not be taken as implying that it is prescriptively good. As noted in footnote D in Chapter 
13, although it is true that how much effort is required to put into reproduction changes the dynamics 
of risk and investment and therefore explains both many observed human dynamics between men and 
women and the (inverted) dynamics of seahorses (where the male broods the eggs in a pouch), 31 we 
should not take this fact as a prescription for appropriate male and female roles in modern society. 
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way to take those interactions and bring them into the laboratory where we can use 
controlled experiments to study them. Early attempts at morality studies were based on 
role-playing games, where confederates would pretend to structure the world in such a 
way as to drive certain behaviors. Two very famous early experiments on morality were 
the Milgram obedience experiment and the Zimbardo prison experiment. 34 

In the Milgram experiment, a subject was told that he or she was to play the role 
of “teacher” while another played the role of “learner.” (The subject thought the learner 
was another subject also participating in the experiment and did not know that the 
learner was an actor.) The two were separated by a wall, so they could not see each 
other. The teacher provided questions to the learner (arbitrary word-pairs that the 
learner had to memorize). The “learners” gave wrong answers to many of the pairs. 
Whenever the learner got it wrong, the teacher was to push a button that would shock 
the learner. As the learner made more errors, the voltage was increased to the point that 
the learner would scream, bang on the wall, beg to stop, and finally, go silent. In reality, 
there were no shocks. The “learner” was an actor playing a part. But the descriptions 
from the subjects (playing the role of “teacher”) make it very clear that they believed 
the shocks were real. 

The Milgram experiment is usually taken as demonstrating the human unwillingness 
to question authority; however, although many subjects continued to deliver shocks to 
the maximal levels, many of those subjects asked to stop and continued only after being 
assured that it was OK, even necessary, to go on. Other subjects refused to deliver shocks 
beyond certain levels. Over the subsequent years, Milgram ran dozens of variations of 
the experiment trying to identify what would make people more or less likely to con¬ 
tinue. The numbers varied dramatically based on the physical and emotional connection 
between the teacher and the learner and on the authority inherent in the experimenter 
assuring the subject that the experiments were necessary. Recent re-examinations of 
the original data suggest that the assurance that the experiments were “necessary” was a 
critical component driving the subjects’ willingness to continue. 35 Most likely, a combi¬ 
nation of both is needed—an authority figure who can convince one that the crimes are 
necessary for an important cause. This is a good description of many of the worst evils 
perpetrated by humans on each other (genocide to “purify the race,” torture to “extract 
information,” etc.). The key to the Milgram experiment may have been an authority 
figure stating that the “experiment requires that you continue” and implying that not 
continuing would ruin an important scientific endeavor. 

In the Zimbardo experiment, two groups of college students were arbitrarily assigned 
roles of guards and prisoners in a mock jail in the basement of a Stanford University 
psychology building. Over the course of less than a week, the guards became arbitrarily 
violent, abusive, and even sadistic. It was so bad that the experiment was stopped after 
only six days—more than a week before it was supposed to. The Zimbardo experiment 
is usually taken as demonstrating how power structures drive dehumanization and 
mistreatment. 

The problem with both of these experiments is that they address extremes and they 
are one-off experiments. They are very hard to replicate, and they are very hard to use 
to get at the underlying mechanisms of human behavior. Nevertheless, both of these 
effects clearly occur in real life. For example, the recent cases of torture in Abu Ghraib 
prison are examples of both the Milgram and Zimbardo effects. (What limited investi¬ 
gations have been done suggest that the soldiers were pushed to create negative-impact 
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situations by interrogator authority figures and then went beyond and reveled in the 
torture they were doing as the moral authority deteriorated. 36 ) 

Although extremely important, the Milgram and Zimbardo studies tell us what we 
fear we already knew—humans are often susceptible to authority (particularly an author¬ 
ity cloaked in a cause), and humans given power will often misuse it. But most people 
don’t live their lives in these extremes. These studies, and the horrors of Rwanda or Abu 
Ghraib, are extremes. Certainly, these situations do occur, but the question with situa¬ 
tions like the Milgram and Zimbardo experiments (or Rwanda, the Holocaust, Nanking, 
Abu Ghraib) is never Are they wrong? They are obviously wrong; the question is how to 
prevent reaching them in the first place. The contributions of the Milgram and Zimbardo 
studies have been When are humans susceptible to authority figures? What makes them able 
to question that authority? (Remember that some subjects in the Milgram experiment did 
question the experimenter and stop the experiment and that other subjects were clearly 
uncomfortable and asked if the other person was OK and continued only after assurance 
from the authority figure that the experiment was necessary. One soldier was uncomfort¬ 
able enough with the situation at Abu Ghraib to leak the photos to the media.) 

The revulsion that we feel at what is clearly evil (abuse of children by trusted author¬ 
ity figures in the Catholic church or the abuse of prisoners [including innocent children] 
by soldiers at Abu Ghraib or the murder of millions of innocents in genocide) identifies 
that evil exists in the world and that humans can do evil, but it doesn’t really help us get 
at the mechanism of that evil. In fact, our institutions have a demonstrated inability to 
deal with evil of that magnitude (e.g., the Catholic church’s inability to come to terms 
with pedophile priests, or the U.S. government’s inability to bring the originators of the 
Abu Ghraib incidents to justice, or the inability of the United Nations to stop genocides 
in Rwanda, Bosnia, or Darfur). The problem isn’t Can we identify evil? We reliably iden¬ 
tify true evil. The problem with true evil is dealing with it. 

Instead, the more difficult question is that of morality in our everyday lives. 31 In our 
daily lives, we are given opportunities to do good and bad—whether or not to give a 
dollar to the homeless person on the street, whether or not to cheat on a test or on our 
taxes. For most people in the world, our normal lives do not resemble either Milgram’s 
or Zimbardo’s horrors. In her chapter in the book Moral Markets, Lynn Stout points out 
that most of our lives are spent in “passive altruism”—we don’t randomly beat people up 
on the street and steal their money. 38 

Some have suggested that this is due to having laws and that people would devolve 
into violent “savages” * 1 without them, for example as depicted in the famous (and deeply 
flawed) book Lord of the Flies by William Golding. But actually, small communities 


H We would also like to get at human questions of fairness and decency without requiring that our 
laboratories devolve into evil Hell-scapes. One of the consequences of the Zimbardo experiment is that 
studying evil is a slippery slope. Part of the reason the experiment was stopped after only six days was 
because of the psychological effect on the guards as much as on the prisoners. 37 

1 1 put the word “savages” in quotes because the term was originally used to refer to non-Western, 
non-highly technological cultures, essentially “barbarians.” The word “barbarian” comes from the Greek 
word describing the unintelligible speech of foreigners (which sounded like “bar bar” to them). 39 The 
important point, however, is that “savages” were not particularly savage, any more than the “barbarians” 
were barbarous (certainly no more than “civilized” people), and they lived in communities with (often 
unwritten) rules that dictated fairness and morality. 40 
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generally do not look anything like Lord of the Flies. They contain neighbors helping 
each other, sharing food, with extensive examples of both passive and active altruism. 41 

When we see our neighbors’ car stuck in the snow (or even a stranger’s car, as hap¬ 
pened outside my house in the big snowstorm we had last winter), we don’t stand 
laughing, we don’t shoot the person and take their car, we all bring out our shovels 
and snowblowers and clear out the car and get the person safely back on their way. 
Firefighters, police officers, and ordinary citizens rush toward danger to help those 
in need. When the 35W bridge in Minneapolis fell during rush hour in August 2007, 
throwing cars into the Mississippi River near the university, our first responders were 
on the scene within minutes, and college students, faculty, and medical personnel from 
the local hospital ran to help. Even from a bridge carrying rush-hour traffic, only 13 
people died that day, due in large part to the remarkable work by the first responders and 
the help they got. 42 Even ruthless pirate ships operated under (written and often very 
detailed) community rules. The Invisible Hook by Peter Leeson gives a vivid description 
of how the “lawless” pirates used carefully constructed internal rules to ensure a viable 
community. We don’t live in a Lord of the Flies situation. So, then, what makes communi¬ 
ties work? If it’s not the laws, what is it? 


Fairness and the creation of communities 

Throughout this book, we’ve seen the importance of breaking the mechanism of 
decision-making into its component parts. We’d like to be able to understand the mech¬ 
anisms of morality in a way that is repeatable and can be studied reliably in the context 
of a laboratory. It would be really useful to be able to study moral questions in such a way 
that we can ask how the brain represents these issues, perhaps by studying them while 
observing brain activity using fMRI. To get at these questions in a more controlled, 
more scientific (and less disturbing) setting, we will turn back to game theory. However, 
instead of trying to determine what the optimal choice is under some normative (pre¬ 
scriptive) assumption, we will examine what choices humans actually make. These 
choices become particularly interesting when we make the games non-zero-sum. 

A zero-sum game is one in which the total wins and losses cancel out. That is, if 
one player wins, then another must lose. In physics, the total amount of energy within 
a closed system cannot be created or destroyed. If two billiard balls hit each other, and 
one speeds up, then the other must slow down. Life, however, is not zero-sum. Many 
human jobs take a tremendous amount of training and practice. I would prefer to pay an 
expert to rebuild my roof so that it will be done right rather than try to learn how to do it 
myself and probably do a mediocre job of it. This is the advantage of specialization. Each 
person becomes better at his or her task, able to produce both more and better products, 
which we can trade between each other, thus increasing the total amount of goods and 
services we have in the system. Our economy has become more efficient) 


1 Although I complained earlier that using efficiency as a sole optimizing criterion is a problem, this 
does not mean that efficiency is a bad thing in and of itself. Specialization can let people concentrate on 
the jobs they prefer while also improving efficiency. There are many things in our modern world (such 
as the laptop computer on which I am writing this book) that would never exist if everyone had to farm 
all their own food and build all their own tools. 
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Thus, we need laboratory experiments in which we have non-zero-sum games. 
Several such games have been developed for studying moral questions. 43 The simplest 
one is the prisoner's dilemma, which introduces the issue of cooperation and defection into 
our discussion. In the prisoner’s dilemma, a pair of thieves have been caught robbing a 
store, but the police can’t find the hidden loot. Without the loot, the police have only 
enough evidence to get them on breaking and entering. But if the police can get one 
prisoner to confess, they can convict the other one. The police tell each one that if he 
confesses and the other doesn’t, they’ll throw the book at the other one and let the 
confessing prisoner off free. If they both confess, then the prisoners will both get light 
sentences (presumably because the police won’t need to go to trial). Basically, we can 
write this as a table with four conditions: if player 1 confesses but player 2 doesn’t, then 
player 1 gets 0 years but player 2 gets 10 years. Similarly, if player 2 confesses and player 
1 doesn’t, player 2 gets 0 years but player 1 gets 10 years. If neither confesses, they get 
1 year. If they both confess, they get 5 years (Figure 23.2). 

The key to the structure of this game is that cooperating 14 when the other player 
defects against you plays you for a sucker and you lose big. However, if both players 
defect, then both players lose. We can write this in a positive form as well as in a nega¬ 
tive one. L Simply put, imagine two players; each one can either “cooperate” or “defect.” 
If both cooperate, both win $50. If one cooperates and the other defects, that defecting 
player gets $80 and the other player gets nothing. If both defect, neither gets anything 
(Figure 23.3). As with the negative version, the key is that if player 1 cooperates, then it 
is better for player 2 to defect than to cooperate, but both players defecting is worse than 
both players cooperating. 

Of course, the key to a community is that we don’t tend to play these games only once; 
we play what is more generally called the repeated (or “iterated”) prisoner’s dilemma. 46 
In this version, the two players keep playing again and again. Imagine two communities 
trading goods between them. Your community grows wheat and the other grows fruit. 
In year one, you bring wheat to the other community and they give you fruit for it. In 
year two, they bring you fruit and you give them wheat for it. Every other year, you put 
yourself at risk, bringing lots of wheat to their community where they have all their 



Prisoner 1 “cooperates” 
(stays silent) 

Prisoner 1 defects 
(confesses) 

Prisoner 2 “cooperates” 
(stays silent) 

Prisoner 1 gets 1 year 
Prisoner 2 gets 1 year 

Prisoner 1 gets 0 years 
Prisoner 2 gets 10 years 

Prisoner 2 defects 
(confesses) 

Prisoner 1 gets 10 years 
Prisoner 2 gets 0 years 

Prisoner 1 gets 5 years 
Prisoner 2 gets 5 years 


Figure 23.2 The prisoner’s dilemma: negative version. 


K The terminology of this traditional example is unfortunate. The prisoners are cooperating with 
each other, not with the police. 

L I started with this in a negative form because it would be kind of weird to talk about the “prisoner’s 
dilemma” with shared money. In any case, throughout the rest of this chapter, I will base these games 
on transfers of money, because that’s how most of the experiments have been done. 41 It is true that 
there are differences in how humans treat positively and negatively framed decisions 45 (see Chapter 3), 
but it is not yet known how these issues interact. 
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people and they could steal it from you and send you home without fruit. Of course, if 
they did that, you wouldn’t come back. This leads to the simple concept of “tit-for-tat.” 

In the early 1980s, Robert Axelrod, a scientist interested in game theory and coop¬ 
eration, ran a series of computer tournaments in which players submitted strategies 
for cooperation and defection. 47 Each player would play a sequence of 200 prisoners 
dilemma games with each other player. (That is, each player would play an iterated 
prisoners dilemma with 200 repetitions against each other player.) What this meant 
was that a player could punish or trick another player. Although lots of very compli¬ 
cated strategies were submitted, the consistent winner was a very simple strategy called 
“tit-for-tat” submitted by Anatol Rapoport, a mathematician studying game theory him¬ 
self. The strategy was to cooperate on the first play, and from then on to do whatever the 
other player did. If the other player started defecting, tit-for-tat would defect, but as long 
as the other player cooperated, tit-for-tat would cooperate. 

One of the things that Axelrod discovered in his series of tournaments is that a com¬ 
munity of tit-for-tat players was remarkably stable. If one simulated evolution among the 
players (that is, the players with the largest wins replicated a larger proportion of that strat¬ 
egy into the next generation, while players with the smallest wins died off), then tit-for-tat 
generally filled out the population and became an evolutionarily stable population—other 
strategies could not break in to the community. Remember that tit-for-tat playing against 
tit-for-tat would both cooperate. Anyone defecting against it would win big once, but 
would then be punished by tit-for-tat and would start losing. Although the tit-for-tat 
player would lose against that one defector, it would still do well cooperating with the 
other tit-for-tat players, but the defector would lose against everybody. 

The problem with tit-for-tat, particularly when playing against another tit-for-tat 
player, is that if there are any errors, then defections begin to propagate through the future. 
Imagine if one time on your way to take your wheat to the other community, you are set 
upon by bandits and all your wheat is stolen. The other community thinks you defected 
and, ignoring all your pleas and explanations, refuses to trade with you again. Axelrod 
found that when errors are included in the game, the optimal strategy was “tit-for-tat with 
forgiveness.” Even among a series of defects, occasionally the player would try a coopera¬ 
tion. It might be just enough to reinstate both players cooperating again. 

The key to all of these issues is that two players cooperating produces more advan¬ 
tage than alternating defections (see Figure 23.3). Strategies in a game world in which 
this is true evolve to produce communities that cooperate. 

Fundamentally, we are social creatures. We live in just such communities. Several 
studies have shown that if you have two groups (think of this as two tribes or two vil¬ 
lages), one of which contains a lot of altruistic cooperators (think of this one as the 
high-tax, high-collaboration village, where people volunteer to help each other out) and 
one of which contains a lot of selfish defectors (think of this as the “I got mine” village, 



Player 1 cooperates 

Player 1 defects 

Player 2 cooperates 

Player 1 gets $50 

Player 2 gets $50 

Player 1 gets $80 
Player 2 gets $0 

Player 2 defects 

Player 1 gets $0 

Player 2 gets $80 

Player 1 gets $0 
Player 2 gets $0 


Figure 23.3 The prisoner’s dilemma: positive version. 
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where people do their best to get their own from the community), then the altruistic 
village will succeed as a group better than the selfish village. 48 The people in the altruistic 
village will spend more time in the cooperate-cooperate entry of the interaction table, 
while people in the selfish village will spend more time in the defect-defect entry of the 
interaction table. 

This is a form of group selection, in that even though the determination of selec¬ 
tive success is completely based on the individual, certain groups do better than other 
groups. Groups that are successful may growlarge enough to split into two groups, while 
groups that are not successful may destroy themselves from the inside. The evolutionary 
implications of this group-selection process is laid out most clearly in the book Unto 
Others, by Elliot Sober and David Sloan Wilson. 

This theory, however, does predict that one is likely to be more cooperative with 
members of ones own village or group rather than between villages or groups. This 
difference is called in-group altruism and out-group xenophobia. In his book Darwin's 
Cathedral, David Sloan Wilson argues that one of the keys to humanity is the flexibility 
and complexity with which we define our groups. He argues that many of our societal 
constructs (religion, nationalism, culture, even sports franchises) are ways of defining 
who is in the in-group and who is not. In Darwin's Cathedral, Wilson explicitly examines 
the development of Calvinism in the 1530s, where exile and ostracism were the main 
tools of ensuring in-group cooperative behavior. 

In his book Historical Dynamics, Peter Turchin defines a term asabiya, which means 
“cohesion” or “group unity.” Turchin argues that the success of dynasties, tribes, 
nation-states, and groups depends on the willingness of individuals to sacrifice for the 
group. Turchin argues that groups, nation-states, tribes, and dynasties all have a cycle, 
where groups arise with a lot of asabiya (in-group cooperation), take over (because they 
are stronger together than other groups that cannot work together), but then lose that 
asabiya over a couple of generations (defectors spread within the group), which makes 
them vulnerable to the next group with asabiya. 

This effect that Turchin is describing occurs because although everyone in the altru¬ 
istic village will do better on average, a selfish person in the altruistic village is going to 
do even better than the altruists in that village. 49 (That person will spend more time in 
the I-defect-you-cooperate entry in the interaction table. Think of the person who rides 
the government-built subway system to protest paying taxes. 50 ) This sets up a conflict 
between levels: at the level of the villages, altruistic villages do better than other villages, 
but at the individual level, selfish defectors within that village do better than the altruists 
in that (mostly altruistic) village. There are two ways to deal with this problem (called 
the free-rider problem 51 ): exile and altruistic punishment. 

Exile entails identifying those defectors and explicitly pushing them out of the com¬ 
munity. One of the major purposes of gossip seems to be the identification of such defec¬ 
tors. 52 Exile forces the person out of the community, thus removing the person from the 
cooperate-cooperate entry in the table. Many groups use ostracism or exile as a devastat¬ 
ing punishment, one that is clearly strong enough to keep communities in line. 53 

A great example of the development of in-group altruism as critical for success on a 
group-by-group basis canbe seen in the development of shipboard rules on pirate ships. 54 
It would be hard to imagine more selfish individuals than pirates, but being trapped on 
a ship together in the middle of the open ocean, facing battle together, certainly defines 
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a very specific in-group. The key factor described by Leeson in The Invisible Hook is that 
of punishment for stealing from the crew or for not pulling ones weight. Not cooperat¬ 
ing with the rest of the crew led to the rest of the crew punishing the selfish pirate, who 
would then change his ways or else. 

Altruistic punishment entails the community identifying and explicitly punishing 
defectors. In a sense, a group of cooperators come together and each sacrifices a small 
amount to ensure that the advantage a defector gains is minimal. 

The simplest game that gets at this question of altruistic punishment is the ultimatum 
game, a laboratory experiment that directly tests a person’s willingness to sacrifice for 
fairness. 55 In the ultimatum game, one player (the donor) gets $20 to split with another 
player (the recipient). The recipient then gets to decide whether to “take it or leave it.” 
If the recipient accepts the distribution, both players get the money, as divided by the 
donor. If the recipient rejects the distribution, neither player gets anything. In a per¬ 
fectly “rational” world, the recipient should take whatever he or she gets. The donor 
should thus give the recipient the minimum amount (say $1 or 1?) and the recipient 
should take it. But if you look at your own reaction to this, you will probably have a vis¬ 
ceral, emotional reaction to being given only $1 or 14 and throw it back in the donor’s 
face. In practice, recipients rej ect anything below about a third of the money, and donors 
(knowing this) divide the money in about a 60/40 split. M 

Interestingly, other animals, even highly intelligent animals like chimpanzees, do 
not seem to show this propensity for altruistic punishment; they take whatever is given 
to them. 58 There are some differences in how the chimpanzees have been run through 
the experiments (they have to run multiple cases because one can’t tell the chimpan¬ 
zee how to play the game). And chimpanzees are sensitive to inequality and unfair¬ 
ness, particularly among compatriots. 59 Nevertheless, there seem to be clear differences 
between humans and our closest genetic relative. Although chimpanzees are our closest 
genetic relative, they have very different social interactions and do not live in the same 
village-based, close-knit communities that humans do. 60 Other animals that do live in 
such communities (such as wolves) do seem to show altruistic punishment. 61 This sug¬ 
gests that the difference maybe less an issue of humanity than an issue of community. 

The point here is that part of our brains demand fairness. Cooperation, trust, and the 
lack of it produce changes in activation of emotional reward circuitry. 62 fMRI results 
have identified that altruistic punishment (rejecting an unfair offer) is associated with 
increases in blood flow in the amygdala and the insula, two structures involved in emo¬ 
tional (Pavlovian) reactions to negative results. In fact, rejection of unfair offers is corre¬ 
lated to the emotion of anger and disgust, and punishment of unfair players is associated 
with the emotion of satisfaction. Punishment of unfair players produces increases in 
blood flow in positive emotional structures such as the ventral medial prefrontal cortex 
and the ventral striatum. The satisfaction of punishing unfair players occurs even when 


M Economists tell us that this is “irrational” in that one loses money by not taking the smaller offer. 56 
Two groups that play more “rationally” than expected (both accept and offer smaller-than-normal 
amounts) are sociopaths, who do not seem able to recognize the other player’s emotional reaction, and 
business-school graduates, who have been trained to act based on their “rational” optimization equa¬ 
tions. 57 Business-school graduates look normal before entering business school but abnormal leaving 
it. I will leave you to draw your own conclusions. 
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the punisher is a third party—that is, player A defects against player B, who cooperated 
(and thus got played for a sucker), and player C spends some of his or her own money to 
punish player A. Players observing an unfair player receiving painful shocks expressed 
satisfaction, an achieved desire for revenge, and showed activity in reward-related areas 
such as the ventral striatum. When we evaluate goals, some of those goals are complex 
and social. 

Variations of the ultimatum game have been examined as well. If the recipient thinks 
the other person is actually a computer program offering choices randomly, that recipi¬ 
ent is more likely to take unfair offers 63 (implying that the rejection of an unfair offer is 
about punishing other players). Donors who think they “earned” the money (through 
some effort, say by performing a task prior to playing the game) offer less than if the 
money is just given to them. The harder the donor worked for the money (the harder 
the task was), the less they will offer. 64iN 

As noted above, altruistic punishment may be the reason for gossip, shame, and rep¬ 
utation. One of the human interactions that I have always found particularly interesting 
is the success of Amnesty International. One of the main things Amnesty International 
does is to write letters to dictators, governments, and human-rights abusers to free dis¬ 
sidents and stop those human-rights abuses. They don’t threaten. They don’t have an 
army. They don’t even threaten with religious “burn-in-Hell” threats. They merely write 
letters—and the remarkable thing is that this often works. 

A similar issue relates to that of forgiveness. As we saw in the tit-for-tat experiments, 
the most resistant strategy to errors and noise is tit-for-tat with forgiveness. Although 
some crimes may be unforgivable, forgiving (possibly after completing an appropri¬ 
ate punishment) is a tool to rehabilitate a defector. If the person cooperates from then 
on, he or she can again become part of the community. In one of the most remarkable 
stories of our time, after the end of apartheid in South Africa, Archbishop Desmond 
Tutu led the Truth and Reconciliation Commission, where the criminals were asked to 
provide a full description, disclosure, and apology. 

The science of morality is still in its infancy. At this point, it is concentrating on the 
descriptive question of What do humans actually do ? The evidence seems to be that we 
have evolved a host of “moral” judgments that allow us to live in (mostly) peaceful com¬ 
munities. These interactions are not always simple and depend on a complex interaction 
that encourages cooperation and reduces the advantages of defection. 

What is particularly interesting is that humanity agrees in large part on morality, 
particularly that of social interactions. The Golden Rule is recognized the world over, 
as part of most religions and as part of most nonreligious moral tracts. Specific prohi¬ 
bitions (gay marriage, abortion, eating pork) vary from religion to religion and from 


N This is why the “it’s your money” anti-tax meme is so effective and insidious: 65 it encourages 
selfish defection from the community. (As Supreme Court Justice Oliver Wendell Holmes famously 
stated, “Taxes are what we pay for civilized society.” This is often restated as “Taxes are the price we pay 
to live in a civilized society”) Similarly, this is why anti-tax groups are trying to get rid of the “withhold¬ 
ing” that occurs in most paid jobs (but not for the self-employed). The more that the anti-tax groups 
can get citizens to see the money as earned and then given away, the less those citizens are willing to pay 
taxes. (Of course, like most selfish defectors in a community, the anti-tax groups want all of the services 
from the community—they just don’t want to pay for them. 66 ) 
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culture to culture, but the primary social and moral interactions (be fair in your dealings 
with others, treat others as you would like to be treated, help when you can) do not. 

When a nonbeliever pressed Rabbi Hillel (110 bce-10 bce) to describe his entire 
religion while standing on one foot, promising to convert if Hillel could do so, Rabbi 
Hillel replied, “That which is hateful to you, do not do to others. All the rest is just com¬ 
mentary. Go and learn.” 67 The Golden Rule is a first step toward cooperation, but it alone 
is not enough. The commentary is long and complicated, and still being discovered. 
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The Conundrum of Robotics 


More often than not, 
we are only trying to know 
that we are not the only ones 
who risk everything for love. 


Freewillincludes three different issues wrappedup in one philosoph¬ 
ical term: (l) the outside perspective (Why do some other beings 
appear as agents with free will to us?), (2) the societal perspective 
(Whom should we blame for criminal actions?), and (3) the inter¬ 
nal perspective (What is the internal sensation that we identify as 
"consciousness"?). By separating free will into these three perspec¬ 
tives, I will show that the first two are not truly issues of “free will," 
and that there is no reason to think that physical machines cannot 
experience an internal sensation of consciousness. 


Free will is a funny term. A pairing of such simple words, it carries with it all that makes 
us feel human. What does it mean to be free? What does it mean to will something? One 
of the great terrors in modern life is that the concept of free will is an illusion. A For, if free 
will is an illusion, then we are just riding a roller coaster, taken to great heights, thrown 
down to great depths, with no control over our journey. But this terror assumes that 
there are only two choices—that either we have complete freedom to choose whatever 
we want or that our feeling of self is an illusion, that we are not in control, that we are 
merely machines, completely predictable in what we do. This dichotomy is the illusion. 

The truth is that we are never completely/ree. We are constrained by the laws of phys¬ 
ics, by our immediate interactions with society, by our physical and monetary resources. 
I can want to fly like a bird all that I want, but no matter how high I jump, I will never 


A The irreverent comic strip Jesus and Mo captured this entanglement elegantly—“Did you know 
that free will is an illusion?” “What about personal responsibility?” “That’s exactly it! Once you accept 
that there’s no such thing as free will, you can do whatever the hell you want!” 1 
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take off and soar into the sky. B I can want to pitch in the major leagues, but I don’t have 
a 100-mile-per-hour fastball. We all accept these limitations, and yet we hold on to the 
term free will. 

Will is an old word, meaning wish or desire. Will is the agency that defines us. 
Fundamentally, it is the fact that the concept of will implies a dualism that lies at the 
heart of the problem of the term free will. If we can explain the reason for an action from 
a source not of our “will,” then we deny our own agency in the taking of that action. 3 We 
will see that this is not a new concept, and that whether or not we are machines is really 
unrelated to whether or not we have free will. 

We need to address free will from three perspectives. From an outside perspective, free 
will is about whether we treat another being as an agent or as a machine. From a societal 
perspective, free will is about whether we can blame individuals for their actions. And from 
an inside perspective, free will is about whether we are truly slaves to our own machinery. 

The outside perspective: The agency of others 

From an outside perspective, we have already seen that we assign agency to many things 
that are not agents. (Remember the discussion of the GPS in Chapter 2?) The assump¬ 
tion of agency is a way for the human brain to handle unpredictability. In particular, 
the assumption of agency allows us to address the complex interaction of predictability 
based on probabilities of action-selection. 

Individuals have personality, character; they are predictable. It is not true that humans 
act randomly. Think for a moment what kind of a world that would be: every person you 
meet could suddenly break into song or could suddenly break into a killing spree. Such a 
world is unimaginable to us. Instead, we talk reliably about propensities, about a persons 
nature. “It’s in my nature,” says Fergus to Dil at the end of The Crying Game,* owning his 
actions, explaining why he took the fall for her. Fergus explains his agency by saying that 
this is within the bounds of what you would expect if you really knew him. Agency requires 
predictability, but only a limited predictability—we identify a character, a nature, a range 
of possibilities, and expect that the agent will act somewhere within that range. 

The problem of mechanics and free will goes back to the time of Galileo, Newton, 
and Descartes, all of whom recognized that physical systems did not have agency in 
the manner we are talking about here—they are completely predictable. (Of course, we 
know now that this only occurs for objects at the macro scale that Galileo and Newton 
were studying. At small enough (quantum) scales, predictability breaks down. 5 Whether 
quantum objects are deterministic or not is still unknown, but our best understanding 
today suggests that they are not; instead, they seem to be probabilistic. As we will see 
later in this chapter, it is possible for quantum effects to expand into the macro world, 
making them fundamentally probabilistic as well. 6 ) Before Galileo, objects themselves 
had agency—Aristotle explained gravity because things wanted to fall to the earth, and 
proposed that objects had both a driving cause, which pushed them into their actions, 
and a teleological cause, which pulled them based on their agency and goals. 7 

Descartes, through his own medical studies, and following directly on Galileo’s 
observations of mechanics, realized that the body itself was a machine. Descartes was 


B Never having been willing to test The Hitchhiker's Guide to the Galaxy’s suggestion to aim at the 
ground and miss. 2 
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an accomplished anatomist and recognized that animals were machines; however, he 
denied that he himself was a machine and exphcitly separated the mind from the body. 8 
Descartes’ concerns lay in predictability—Newtonian mechanics imphes that if you 
know enough details about the state of the universe at any moment, it is completely pre¬ 
dictable from then on. Sometimes called the “clockwork universe” theory, the remark¬ 
able success of Newtonian mechanics in predicting motion in the physical world led to 
the hypothesis of determinism. 9 Philosophers at the time believed that there were only 
two choices—an extra-universe decision-making agent with free will, or that everything 
is predetermined, and all free will is an illusion. 

Some philosophers have tried to rescue the randomness needed for free will from 
quantum mechanics. 10 The best current physics theories suggest that the universal deter¬ 
minism implied by Newton is wrong—we cannot predict the actions of a quantum par¬ 
ticle, it is truly random (within the constraints of probabilities 0 ). 11 Neuroscientists have 
tended to be dismissive of theories that base free will on quantum mechanics because 
quantum randomness does not generally reach to the macro level that we experience 
in our everyday world. In large part, this occurs because quantum randomness is con¬ 
strained by probabilities. With enough elements, the probability distributions fill out 
and things become extremely predictable. Even products that depend on quantum prop¬ 
erties (the laser, the transistor) provide predictable results that we depend on daily. 12 
However, nonlinear mathematics provides a mechanism through which small (micro) 
fluctuations can drive large (macro) events. 13 In these equations, small changes perco¬ 
late through the system, producing a large divergence in final outcomes. 13 Individual 
neurons are highly nonlinear, containing many positive feedback functions that could 
carry these micro-level effects into the macro-level world of neuronal function.® 


c One of my favorite quotes is one I first heard from my father (who is a theoretical physicist): 
“Electrons have free will.” We can predict very accurately what electrons will do in the aggregate. We 
can provide a probabilistic prediction about what a single electron will do, but it is impossible to say 
with certainty what a single electron will do. 

D This growth from micro-level manipulations to macro-level effects is sometimes called the “butterfly 
effect,” from the concept that how a butterfly flaps its wings in China (a small change) can change the path 
of a hurricane in the Caribbean (a large effect). It was first described by Henri Poincare (1854-1912) in 
the late 1800s but was hard to really see until the development of the computer allowed reliable computa¬ 
tion of complex equations. The first computer demonstration that small changes led to large divergences 
under certain conditions was from a meteorologist named Edward Lorenz (1917-2008), who was trying 
to simulate weather patterns in the 1950s. His simulations cycled through a series of simple differential 
equations. Each cycle, the simulation printed out what should have been a complete description of the 
system at a given time. When his computation would stop (due to electrical problems with the old com¬ 
puters of the time), he would retype in the printed numbers. This was a completely deterministic system, 
but what he found was that differences in the least significant digits would percolate up into the most 
significant digits and produce highly divergent effects. For those interested in this effect, James Gleick’s 
1987 popular science book Chaos remains one of the best introductions to the field. 

E Cells have many individual components that work at quantum levels. One of the most straight¬ 
forward examples is that of ion channels. These ion channels are a key component of cellular func¬ 
tion, including neuronal function. Ion channels are made up of intricately shaped proteins that form 
mechanical devices that allow certain ions to pass through the cellular membrane while restricting 
the flow of other ions. The distribution of ions—which are inside the cell and which are outside the 

( continued ) 



230 


THE HUMAN CONDITION 


Identifying agency in others comes down to predictability in the aggregate but 
unpredictability in the specific. Thus, one has an identifiability of personality and char¬ 
acter (“It’s in my nature”) and yet an inability to know exactly what an agent will do 
in a specific case. Quantum randomness (particularly when combined with nonlinear 
chaotic systems) can provide the necessary constrained randomness to give us what we 
observe as agency in others. But all that is really required for us to find agency in another 
is a predictable randomness. Whether that comes from quantum randomness or sim¬ 
ply from an inability to know enough details to predict accurately we reliably use the 
concept of agency to describe randomness occurring within a shape of possibilities. 

It is important to differentiate these points I have made here from popular books on 
quantum theories of mind such as Roger Penrose’s The Emperors New Mind. Penrose 
argues that machines without quantum mechanics cannot be intelligent because quan¬ 
tum mechanics provides a special component to our brain function. Penrose’s proposal 
is just a new version of Descartes’ error—identifying a conduit from a nonphysical self 
to the physical. Descartes identified that connectivity with the pineal gland, fully aware 
that all mammals have a pineal gland; Penrose identifies the quantum components as 
the special key. The problem is that all matter is quantum and that all animals have quan¬ 
tum components in their neurons. F What I am saying here is that there is probabilis¬ 
tic randomness in the other creatures we interact with (whether it be from quantum 


cell—controls the cells function, including firing a spike (carrying information) and releasing neu¬ 
rotransmitter (communicating to another cell). Each ion channel has a number of different configura¬ 
tions; each configuration has different propensities to be open or closed, to allow ions to flow through 
or not. Whether an ion channel is open or closed at any moment is a random occurrence. 14 

Measures of each individual ion channel show that while the likelihood of an open channel clos¬ 
ing or a closed channel opening has a characterizable probability, the actual timing of each is random. 
Because a cell typically has thousands of ion channels of a given type, performing a specific function, 
the individual randomness averages out and the macro level of the cellular function is extremely con¬ 
sistent from event to event. 

But there are some cases where the number of channels is actually very small, where feedback pro¬ 
cesses can translate quantum micro changes into macro changes. One of those situations is in the firing 
of action potentials. Neurons integrate information and then send that information on by firing action 
potentials or spikes. (See Appendix A.) The spike itself is a positive feedback from voltage-sensitive 
sodium channels. 15 This means that the threshold between spiking and not can depend on a very small 
number of extra sodium channels opening up. 

Computational studies and experiments that have replaced quantum-random channels with persis¬ 
tent channels have found that there are macro-level network properties that vanish without that quantum 
randomness. (For example, cells in the entorhinal cortex show an inherent oscillation at the theta rhythm 
of 5-7 Hz. This oscillation depends on channel noise and vanishes without the randomness inherent in 
these channels. 16 ) Other examples in which stochastic randomness at the micro (quantum) level can be 
reflected in macro-level effects include whether or not a synapse releases neurotransmitter, and how a 
neurons axon reaches out to find other neurons during development (which relates to how the network 
is wired up). 17 But, as with transistors, lasers, and other phenomena dependent on quantum properties, 
macro-level effects in cells (such as spiking) are highly reliable phenomena. 18 How much quantum ran¬ 
domness affects subtle properties in neuroscience (through the butterfly effect) is still unknown. 

F In fact, the quantum components (microtubules) that Penrose and his colleague Stuart Hameroff 
reach for 19 are in every cell in every multicelled creature from the smallest worm to the largest whale. 
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randomness or chaotic indeterminacy) and that randomness leads us to assign agency 
to others. As we have already seen, humans assign agency to anything they interact with 
that is unpredictable within a limited range (such as fire or weather); I am making no 
claims about the consciousness of those other agents. 

So we can identify why other people appear to be agents to us—they are predictably 
random. But I suspect that readers are unlikely to be satisfied with this story because 
I have sidestepped the two real problems with free will. First, How can we hold people 
responsible for their actions if they don’t have free will? and second, Do you, dear reader, 
have free will? 


The societal perspective: The question ofblame 

Ethicists and legal scholars have come to the conclusion that someone forced to do 
something is not at fault. 20 Originally, this came from the concept that if someone holds 
a gun to our head and says, “commit this crime,” then we are not at fault for the crime, 
because we are not the agent that initiated the action. 0 

The problem, however, arises when we start to have explanations for why someone 
committed a crime. If we find that someone has a brain tumor that impairs the process¬ 
ing of his prefrontal cortex, do we blame the person for the action? What if the person 
has an addiction? What if we come to the point that we can explain the neuroscience of 
decision well enough that we can say that anyone with a brain like this would have made 
that choice? Do we forgive the criminal the crime then? 

It is important to note that this concept predates neuroscience by millennia. In every 
era, there have been explanations for why someone does something that does not imply 
their responsibility. Before science, there were witches and devils. 21 Before neurosci¬ 
ence, there were societal and developmental explanations. 22 

This question of our ability to predict criminal behavior also opens up an entirely differ¬ 
ent set of problems—if we can absolutely predict that someone will commit a crime, aren’t 
we obligated to stop him? 23 What if we can only predict that he will commit the crime with 
a high probability? What is the balance between that probability and the persons rights 
to freedom? Most modem legal systems have decided to side on the balance of freedom, 
that one has to give the person the opportunity to not commit the crime, that it is unfair to 
judge a person for his or her thoughts and only fair to judge a person for his or her actions, 
and thus that one has to wait for the crime to be committed before acting. 11 

In large part, the problem of the ethics of choice lies in the distinction that has been 
drawn between the goal of punishing the responsible agent and the goal of ensuring that 
dangers to society are successfully removed from that society. As we saw in the last two 
chapters, one of the key factors that makes us human and has allowed us to live in stable 


G This is not entirely true. For many of these situations, there is a choice, but the other option is so 
heinous that the crime becomes the best choice. Perhaps instead of saying that the person committing 
the crime is “not at fault,” we should say that we are merciful, and forgive the action, because we under¬ 
stand the limitations of the options available. 

H Of course, there’s no reason one can’t prepare defenses beforehand. Locking your door is a very 
different response to preventing theft than locking up the potential thief. 
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and complex societies is our willingness to take part in altruistic punishment, in which we 
sacrifice some of our own resources to punish a wrongdoer. The effect of this is to make 
it less worthwhile for wrongdoers to do wrong, which helps enforce cooperation. 

In small groups, where everyone knows everyone else, where an individual is highly 
dependent on the willingness of the rest of the tribe to work with that individual, coopera¬ 
tion is reliably enforced by altruistic punishment (see Chapter 23). However, as societies 
get larger, it becomes more and more difficult for altruistic punishment at an individual 
level to produce sufficient incentive to enforce cooperation. It becomes too easy for an 
individual to cheat and then walk away and vanish into the city. In situations where there 
are two rival groups, it becomes too easy for altruistic punishment to devolve into a fight 
between groups and to escalate into what is effectively a tit-for-tat gang war. (The unfor¬ 
tunate flip side of in-group altruism is out-group xenophobia.) The legal system shifts the 
inflictor of the punishment from the individual to the society itself. 

The real goal of a modern legal system is to ensure the ability of a society to function. 
However, although the medical community has spent the past several decades fighting to 
ensure that only scientifically-tested therapies get used, a process called “evidence-based 
medicine,” no such constraint yet exists in the legal system. 24 This means that laws are 
sometimes based on simple conclusions from prescriptive moral principles rather than 
descriptive measures of reality. Two clear examples of this are the surprising decisions 
to implement austerity measures in response to shrinking economies, and the legal 
decisions to criminalize rather than to treat drug use. 25 

Much of the ethical argument is based on whether or not it is “fair” to punish some¬ 
one, rather than how best to maintain societal function. One of the best examples of this 
is the “war on drugs.” 1 No one denies that drug use is a problem. As we saw in the addic¬ 
tion chapter (Chapter 18), drug use leads to clear diminishments of well-being, both 
within the individual and within society. However, drug use, even highly addictive drug 
use, particularly early on an individual’s path, is often a nonviolent individual problem 
rather than a societal problem. 27 Extensive data have shown that sending nonviolent 
addicts to treatment reduces the addiction, reduces the likelihood that they will turn 
violent, and has good success rates bringing them back to society. In contrast, sending 
addicts to prison increases the likelihood that they will turn violent, has poor success 
rates bringing them back to society, and is more likely to lead to relapse, often with worse 
drugs, on return to society. And yet, the discussion within American society today has 
clearly settled on criminal punishment and prison for individual drug use. 

I will never forget an example of just such a disconnection, which I heard at my college 
graduation. The speaker was the novelist Tom Clancy, who was trying to pander to engi¬ 
neers in a speech at the engineering school graduation ceremony. Early in the speech, he 
talked about how important it was to look at the scientific data, and how sometimes the 
scientific data were different from common sense, so engineers had to learn to look at 
the data itself. Not ten minutes later, in the same speech, Clancy denounced drug addic¬ 
tion treatment centers, saying that he didn’t care if the data said that treatment worked 
better than prison, the addicts knew they would go to jail and decided to take the drugs 


1 As has been pointed out by many commentators and philosophers, the terminology “war on 
drugs” enhances certain perspectives on handling the problem and likely limits our political options in 
handling the problem of addictive drug use. 26 
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anyway, and had to be sent to prison. These logical disconnects lie in the limited view 
of free will ensconced within our legal system as an all-or-none phenomenon: either 
someone is free to choose and should be punished, or someone is a slave and the actual 
agent of the decision needs to be found instead. As we’ve seen throughout this book, 
the story isn’t so simple. 

The legal system has had problems with the question of blame and punishment long 
before the problem that understanding the mechanism of decision-making can lead to 
explanations independent of free will. 28 However, if we move away from the question of 
“blame” and toward goals of mercy and forgiveness at the individual level, while maintain¬ 
ing the goal of keeping a working society at the societal level, we find that the question 
of free will again becomes moot. 

Of course, as we discussed in the first part of this chapter, free will means we can pre¬ 
dict in the aggregate but not in the individual. 29 This means that the prediction of future 
action is only that: a prediction. As shown in the movie Minority Report, it is inherently 
dangerous to assume that a prediction is reality. 30 There is a big difference between lock¬ 
ing your door to prevent theft and locking up the thief before he breaks into your apart¬ 
ment (Footnote H, above). 

We can understand the reasons why someone has done something and rather than 
worrying about whether to blame them for their actions or not, we can worry about 
how to ensure that it does not happen again. Punishment is only one tool in that tool¬ 
box; there are many others. Knowing which ones will work when and how to imple¬ 
ment them are scientific questions that depend on our understanding of the human 
decision-making machine. 

Two great examples of using other tools are the successful uses of shame by Mahatma 
Gandhi and Martin Luther King. 31 They were able to achieve their aims by showing the 
world what others were doing. (Do not underestimate the importance of television and 
movie newsreels in their successes.) By using the human intersocial mechanism that 
recognizes strength not just in terms of physical ability but also in terms of “heart,” of an 
ability to remain standing even in the face of greater physical power/ Gandhi and King 
were able to accomplish great things. 

So a random process combined with a mechanism to ensure that the randomness 
falls within a probability distribution is sufficient to explain what we observe as agency 
in others. And recognizing that altruistic punishment is only one tool in society’s tool¬ 
box to ensure cooperation rather than defection leaves moot the ethical question of 
whether someone actually has free will or not. However, we have so far avoided the hard 
question of consciousness —Do you, dear reader, have free will? 


The inside perspective: Consciousness 

We all have the experience of making decisions but not knowing why we made them. We 
all have the experience of not paying attention and looking up to recognize that we’ve 
done something we would have preferred not to, whether it be letting our addiction, our 


J 

fight. 


Rocky is the hero of his movie because he remains standing at the end, even as he loses the 

32 



234 


THE HUMAN CONDITION 


anger, or our lust get the better of us, or whether it be that we stopped paying attention 
and found ourselves having driven our friend to our office instead of the airport. This 
led to the rider-and-horse theory—that there is a conscious rider and an unconscious 
horse. The rider can steer the horse, but the rider can also release control, allowing the 
horse to take its own path. 33 Sometimes the horse takes a sudden action that the rider 
then has to compensate for. K 

Similarly, we have all had the experience of rationalizing an action that was clearly 
taken in haste: “I meant to do that.” Following the famous Libet experiments, which 
showed that conscious decisions to act are preceded by brain signals long before the 
time we believe that we made the decision, many philosophers have retreated to the 
hypothesis that consciousness is wholly an illusion and that the rider is but a passen¬ 
ger, watching the movie unfold. Others have suggested that consciousness is more like 
a press secretary than an executive, and that consciousness entails rationalizing the 
actions that we take. Daniel Wegner differentiates actions that we identify as voluntary 
and those that we do not, and notes that there are both cases of (l) actions we do not 
have control of but believe we do and (2) actions we do have control of but believe we 
do not. Wegner explicitly identifies “free will” with the illusion of agency. However, 
Libet’s experiments are based on the idea that the conscious decision to act occurs 
when we recognize (linguistically) the decision to act. Libet and others have suggested 
that consciousness is monitoring the actions we take so that it can stop, modify, or veto 
the action. 34 

The problem, as I see it, is that philosophers have the concept that consciousness 
must be an external thing, external to the physical world, or an emergent property of 
physical things that is (somehow) fundamentally different from the underlying physi¬ 
cal things. 35 This issue first came to the fore with Descartian determinism, following on 
the implications of Galileo and Newton and Johannes Kepler that one could accurately 
predict the motion of objects, and that they did not need agency to move. 1 If the planets 
did not need agency to act, then neither did the machinery of the body. 38 Clearly desper¬ 
ate not to abandon human free will, Descartes concluded that although animals were 
machines, he himself was not. 


K The multiple decision-making system theory suggests why we have multiple (sometimes con¬ 
flicting) desires. Sometimes the Pavlovian system wants one thing, while the Deliberative wants 
another... all while the Procedural system is taking actions on its own. 

L Compare Thomas Aquinas (1225-1274), who hypothesized that angels pushed the planets in 
their orbits. 36 Although angels are not taken seriously as an explanation for planetary motion anymore, 
it was a serious theory at the time. Aquinas believed that things could not move without agency. With 
the breakthroughs by Galileo, Descartes, Newton, Brahe, and Kepler, agents were no longer needed to 
move the planets, and the angel-pushing theory was abandoned. 

The timeline is actually that Galileo (1564-1642) precedes Descartes (1596-1650) who pre¬ 
cedes Newton (1643-1727). Galileo showed that one could measure physical reality experimentally. 
Descartes showed that space could be represented mathematically. And Newton derived accurate 
mathematical descriptions of deterministic motion. The two other key important figures in this time¬ 
line are Tycho Brahe (1553-1601) and his student Johannes Kepler (1571-1630), who showed that 
the planets moved in regular orbits, which could be described as ellipses. As with modern science, 
these men all knew each other s work well, read each treatise as it came out, and built on them. None 
of them worked in isolation. 37 
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Descartes’ famous line is, of course, cogito ergo sum (I think, therefore, I am), which is 
fine but says nothing about whether his fellow humans think or whether the language 
they spoke to him merely gave him the illusion that they thought, and says nothing 
about whether his fellow animals think, even though they cannot tell him their intro¬ 
spections. Descartes concluded that the pineal gland is the connection from an external 
conscious soul to a physical being. While the pineal gland does connect the brain to the 
body, M all mammals have pineal glands. 40 An accomplished anatomist, Descartes was 
well aware of this yet concluded that only the human pineal gland “worked” to connect 
souls to bodies. 41 

A more modern version of this desperate hypothesis can be found in the quantum 
theories of Roger Penrose and Stuart Hameroff, who have argued that consciousness 
cannot arise from a deterministic system and thus that it must depend on quantum 
fluctuations. 42 In particular, they have argued that the mind interacts with the brain 
by manipulating quantum probabilities—free will occurs because our separate souls 
can manipulate the quantum probabilities to give us free will. Notice that Penrose and 
Hameroff s theory that an external soul controls quantum probabilities is very different 
from the earlier discussion of quantum probabilities as a potential source of random¬ 
ness in others. I suspect that if we were actually able to control quantum probabilities, 
we would, as shown in Neal Stephenson’s Anathem, actually be much better at certain 
problems (like code-breaking) than we are. This quantum theory is often stated as a 
statement along the lines of Machines cannot be conscious. Since I am conscious, I am not a 
machine, 43 which is a thinly veiled restating of Descartes’ cogito ergo sum. 

My take on this is that the problem is, when we imagine it, we believe that there 
are only two choices—either we are conscious beings who can take any action or we 
are slaves who have no choices. But like Fergus and Dil in The Crying Game, we have 
personalities, and we are true to our “nature.” 

The word robot comes from a play by Karel Capek (titled R.U.R. [Rossum’s Universal 
Robots]), which was really about the effect of industrialization on individuality. 
Similarly, our more modern concept of robots comes from observations of highly com¬ 
plex machines, many of which are used in industrialized factories. It speaks to our fear 
that robotics leads to slavery, but as we’ve seen, the issue isn’t slavery. Even if conscious¬ 
ness were to be an illusion, it wouldn’t change our perception of our consciousness. It’s 
not that we are suddenly going to become slaves—either we are already or we aren’t. N 

The world is more complex than that. Certainly there are decisions that are made 
without conscious intent, but there are also very long conscious deliberations that take 
minutes, hours, days, weeks, or even months. Even in the case of specific decisions that 
occur too quickly to be conscious, we can decide to train ourselves to make the right deci¬ 
sions at the right time. It takes time to learn to ride a bicycle. Both sports stars and soldiers 
know that doing the right thing at the right time quickly (without thinking) takes lots of 
practice. Although the action at the moment may be reactive, the decision to practice is 


M The pineal gland releases hormones into the bloodstream based on the firing of neurons and 
detects hormone levels in the bloodstream, translating neural representations into hormone levels and 
hormone levels into neural representations. 39 

N I have a friend who likes to say that it doesn’t really matter whether you have free will or not. If you 
do, then you better believe you do. If you don’t, then you’re predetermined to believe that you do. 
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often a conscious one. In cases where we decide to actually get up and go to the anger 
management seminar or the addiction treatment center, the conscious decision isn’t to 
not be angry or to not take drugs; the conscious decision is to retrain ourselves. 

A more nuanced discussion has appeared in the science fiction literature of the past 
several decades—with Star Trek: The Next Generation, as the android Data tried to 
become human; with Blade Runner and its “replicants”; and recently with the newly 
reinterpreted Battlestar Galactica, in which the humans have to decide whether the 
Cylons have sufficient free will to be treated as human. 

In many of these examples, the journey for the robot to become human is based on 
the belief that the robots need to feel emotions. A recurring theme in Star Trek: The Next 
Generation is Data’s search for emotions. Yet, throughout, he is clearly as human as the 
rest, with foibles, errors, and decision-making abilities. In Blade Runner, the replicants are 
identifiable only because they do not have the correct emotional reactions to shocking 
images. But a theme in the movie is the question ofwhether Rick Deckard, the hard-bitten 
human detective hunting the replicants, would pass the emotional test for humanity if he 
were to take it. (This theme is only implicit in the movie but is something Deckard wor¬ 
ries about explicitly in the Philip K. Dick novel Do Androids Dream of Electric Sheep?, 
from which Blade Runner derives.) In the end, the movie hinges on a decision made by 
the lead replicant Roy Batty, who decides not to throw Deckard off the building ledge 
and instead to lift him back up. Deckard wonders at the reasons for the decision but never 
doubts that the replicant made a decision of his own free will that saved Deckard’s life. 
Similarly, throughout the new Battlestar Galactica series, the Cylons make clear decisions, 
and while the humans debate whether to treat the Cylons as humans or machines, they 
never doubt for a second that Cylons are dangerous decision-makers. 

The problem with all of these discussions is that we haven’t (yet) built machines 
that have free will, and we don’t know what it would be like to build such a machine. 
Obviously, there are many machines that do not have free will. But that doesn’t preclude 
the possibility of a machine with free will. It doesn’t preclude the possibility that we are 
living machines with free will, that we are machines who fall in love, robots who write 
poetry, that we are physical brains capable of making our own decisions. 

Books and papers for further reading 

• Antonio Damasio (2010 ). Self Comes to Mind: Constructing the Conscious Brain. New 
York: Pantheon. 

• Jeffrey Gray (2004). Consciousness: Creeping up on the Hard Problem. New York: 
Oxford University Press. 

• Robert Kurzban (2010). Why Everyone (Else) is a Hypocrite. Princeton, NJ: Princeton 
University Press. 

• Neil Levy (2007). Neuroethics. Cambridge, UK: Cambridge University Press. 

• Daniel M. Wegner (2002). The Illusion of Conscious Will. Cambridge, MA: MIT 
Press. 


EPILOGUE 


We’ve taken quite a long journey in this book, from our first discussions of decision¬ 
making and the sense-evaluate-act cycle of the thermostat to imagination, morality, 
and how we are machines with free will. In this book, we’ve explored the science of 
decision-making, and how humans (and other animals) make decisions, sometimes 
consciously, sometimes less so. We like to think we are rational creatures, but we often 
make irrational decisions. This irrationality comes from the separation between the 
Deliberative “self” and the Pavlovian and habit-driven (Procedural) “other,” but those 
other decision-making systems, the Pavlovian action-selection and motivation systems, 
the situation-recognition (memory) components, the habits learned over years, all make 
up the person you are. To paraphrase Jonathan Haidt’s horse-and-rider analogy, 1 you are 
both the horse and the rider. These are the stories we tell ourselves, the memories that 
define for us who we are. 

Although this is not a self-help book, the theory that the decisions you make arise 
from multiple, interacting decision-making systems has implications for how we can 
change our lives if we desire to. The reason that some decisions are made by emotional 
(Pavlovian) systems, while others are reactive action-chains (Procedural), and others 
are taken only after extensive consideration of the options (Deliberative) is that each 
of these systems has advantages and each has disadvantages. The Deliberative system is 
slow to execute (it takes time to consider those options) but flexible (considering leaving 
your job doesn’t mean you will). The Procedural system is really good for actions that 
need to happen the same way every time quickly (like playing a musical instrument). 
Emotional systems are critical for social interactions. 

If you could steer yourself from one system to another at the right time, you could 
improve your decision-making. For example, engaging the Deliberative system makes it 
easier to reject impulsive, emotional desires. Contingency Management is a technique 
where one concentrates on a concrete alternative reward as a means of behavioral modi¬ 
fication. This seems to have the effect of engaging Deliberative over emotional or reac¬ 
tive systems. 2 

As another example, disengaging the Deliberative system and allowing the Procedural 
system to run is critical for appropriate performance in sports and other behaviors that 
require repetitive behaviors. My favorite example of this is how Annie Savoy (Susan 
Sarandon) distracts the pitcher Ebby LaLoosh’s (Tim Robbins’) Deliberative system by 
providing him women’s lingerie to wear in the baseball movie Bull Durham . 3 Once his 
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(well-trained) Procedural system is given free rein, he discovers that he can pitch strikes 
reliably. Of course, this only works if the Procedural system has been properly trained. 
Although we do not generally have conscious control of Procedural behaviors during 
execution of that behavior, the decision to practice is often a conscious one. 4 As every 
musician knows, the key to playing a piece well is practicing it. A 

Similarly, the emotional, Pavlovian systems are critical to making appropriate deci¬ 
sions. 5 Courage is not the absence of fear, but the overcoming of fear. 6 One should be 
afraid of a burning building. And, as we saw in Chapter 23, these emotional reactions 
are critical to appropriately navigating the social milieu we live in. Altruistic punishment 
(the willingness to sacrifice some of one’s own gain to punish a cheater) occurs because 
we feel emotional responses to unfairness. 7 

Not only is it possible to train each of the systems individually, it is possible to learn 
how to steer oneself between systems, learning when to deliberate and when to allow 
the Procedural system to run. 8 In addition, changes in the support structures (such as 
the situation-recognition system, Chapter 12) can change ones decisions. 9 For exam¬ 
ple, realizing that there is “no such thing as one drink” for an alcoholic means that the 
choices are not between the boring Dr. Jekyll and a suave man-about-town, but rather 
between Dr. Jekyll and Mr. Hyde. This changes one’s decisions. 

Our understanding of how we make decisions has enormous implications for how 
we understand ourselves, how we interact with others, how we see the world around 
us, and how we define and treat mental dysfunction. Although a lot is known about 
the decision-making system and how the brain creates the mind, there is a lot that is 
still unknown. Most of the concepts that we’ve discussed in this book are still areas of 
very active research within the scientific community. There are a slew of open questions 
still unanswered. How do these multiple systems interact? What controls which system 
drives behavior when the multiple decision systems are in conflict? How does the tran¬ 
sition from Deliberative to Procedural (habit) occur? What is self-control, really? How 
does changing our conceptual narrative change us? How does the Pavlovian motivation 
system affect the Deliberative and Procedural (habit) action-selection systems? What 
is the role of attention in evaluation, particularly during deliberation? What does treat¬ 
ment do in addiction or the other psychiatric ailments? What is going wrong in these 
psychiatric ailments ? How do we ascertain the underlying causes? Can we learn to guide 
each individual patient to the right treatment? There’s a lot of work still to do here. What 
are we going to find next? I don’t know. But I’m sure it’s going to be a great ride. 

A. David Redish 
Minneapolis, Minnesota, 2010-2013 


A Often the push to practice is driven from the outside, as every parent cajoling a child to practice 
a musical instrument, every drill sergeant driving new recruits, and every advisor pushing graduate 
students knows. 



APPENDIX 


In these three appendices, I turn to issues of neuronal function (A), represen¬ 
tation (B), and memory (C). Within each of these, there are interesting com¬ 
putational issues that bear indirectly on decision-making processes, but the 
complexity of which is tangential to the main theme of the book (how we make 
decisions). 
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Appendix A 

INFORMATION PROCESSING IN NEURONS 


dancing sunlight on the path 
reveals flashes of information 
leaves dancing in the morning breeze 


It is often said that the brain is an "information-processing device," 
that it takes information in from the world, transforms it into an 
internal representation, and takes actions that change the world 
around it. Even if the brain does process information, the brain 
is not a computer like the computer on your desk anymore than 
the swan we saw in Chapter 2 is an airplane. In this chapter, I 
will provide a starting discussion of cells, neurons, and information 
processing therein. 


First of all, the brain is made of cells. Each neuron has all of the biological material 
needed for normal cellular function. Each individual neuron in the brain has a nucleus 
and shares the same DNA blueprint with every other cell in your body. Each neuron has 
all of the machinery to translate that DNA blueprint into RNA and then into proteins. 
Second, a neuron is an information-processing device. It has a tremendous amount of 
intracellular (and extracellular) machinery that reacts to changes in such a way as to 
calculate information. 


Information 

What is information ? In the modern world, we encounter information processing 
every day. A The mathematics of information that enabled the computer revolution 
(sometimes called the information revolution) was worked out in the 1940s by Claude 


A In T he Amazing Story of Quantum Mechanics, Jim Kakalios argues that the technologies that 
changed our world in the 20th century are all breakthroughs in information processing. 
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Shannon/ an engineer working for Bell Labs/ building on work he did in cryptography/ 
both on his own and in collaboration with Alan Turing during World War II. 2 

Mathematically, Shannon realized that “information” was about separating possible 
codes into groups/ Shannon characterized this as answering yes/no questions. Each 
answer to a yes/no question equaled one bit of information. In the modern world, we 
encounter this definition of information every day. Computer memory and hard drive 
capacity are generally measured in eight-bit units called “bytes” (as in megabytes [one 
million bytes] and gigabytes [one billion bytes]). A byte consists of eight small physical 
things that can each be in one of two states (on/off, yes/no). In the memory of your 
computer or in a USB flashstick, that small physical thing is usually a tiny magnet that 
can be flipped back and forth. On a CD, that small physical thing is a miniscule pit dug 
into the plastic that reflects light of different colors depending how deep it’s been dug. 

Shannon realized how to use probability theory to mathematically define less than 
one bit of information. The game Twenty Questions is based on this concept. If birds fly 
and mammals walk, then answering the question Does it fly ? provides information about 
whether the person is thinking of a bird or a mammal. Of course, some birds don’t fly 
(think penguins) and some mammals do (think bats). Since Does it fly? mostly, but not 
completely, separates birds and mammals, the answer to Does it fly? provides less than 
one bit of information about whether the person is thinking of a bird or a mammal. 
Decision-making is about changing our actions based on the information available to 
us in the world. 


Neurons 

Neurons (like all cells) maintain a voltage difference between the intracellular and extra- 
cellular space. There are more negatively charged ions inside the cell than outside, which 
creates a voltage difference between the inside and the outside of the cell. In effect, a 
neuron is like a battery, maintaining a voltage across the cellular membrane. This bat¬ 
tery is maintained through small holes in the membrane called ion channels . E These ion 


B Bell Labs was the research department for the telephone monopoly AT&T. For those too young 
to remember, the landline telephone system in the United States was originally controlled by a monop¬ 
oly company AT&T. It was broken up into many little companies in 1984. 

c The best description of information theory and its role in cryptography that I’ve ever seen is in 
Neal Stephensons Cryptonomicon, in which the Allies are trying to determine how to use the German 
and Japanese codes that they’ve broken (which they broke using the nascent information theory) in 
such a way that the Germans and Japanese do not realize that the codes have been broken. 

D The concept of “information” turns out to be identical to the thermodynamic concept of entropy, 
and thus the amount of information needed to describe a system is termed the “entropy” of the system. 
The physics of entropy, information theory, and thermodynamics is beyond the scope of this book, but 
there are many good books on the subject. I recommend Charles Seife’s Decoding the Universe, Murray 
Gell-Mann’s The Quark and the Jaguar, or Heinz Pagels’ The Cosmic Code or his Perfect Symmetry. James 
Gleick’s new The Information, A Theory, A History, A Flood is an excellent discussion of the history and 
meaning of information theory. 

E Actually, ion channels are not really holes so much as they are specialized pumps. Each channel 
will only permit certain ions through. Thus, there are potassium channels, which only allow potassium 

( continued ) 
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channels are made up of intricately shaped proteins that form mechanical devices that 
allow certain ions to flow through, while restricting the flow of other ions. Proteins are 
made up of long chains of amino acids following the blueprint in the DNA, translated 
through the mechanisms of RNA. 

These ion channels are little tiny mechanical machines. While some of them are con¬ 
stituency open, always allowing their specific ions to flow through, others can open and 
close. For example, one channel type, the voltage-gated potassium channel, literally has 
a ball of amino acids on a chain of amino acids attached to the base of it. When the 
voltage in the cell changes, the chain changes shape and the ball pops into the channel, 
blocking it. As another example, the voltage-gated sodium channel twists like an iris so 
as to open or close depending on the membrane voltage. There are actually many sub- 
types of voltage-gated potassium and voltage-gated sodium channels, each with slightly 
different properties. These properties change the dynamics of the neuron to make it do 
the information processing it does. This step from which ion channels are expressed 
where on the neuron to how that neuron processes information is a very active area 
of research in neuroscience today. Some of these steps are known, some are still being 
investigated. 


Information processing in neurons 

The key to information processing in the brain is a phenomenon called the action poten¬ 
tial or (more colloquially) the “spike.” A spike is a short event in which specific channels 
open up allowing positive ions to rush in, which changes the cross-membrane voltage; 
then the first set of channels starts to close, and a second set of channels opens up, which 
brings the voltage back down. This entire process takes a few thousandths of a second 
(a few milliseconds). A cell integrates changes in its cross-membrane voltage and fires a 
spike when that voltage gets large enough (crosses a threshold). 3 

Following Shannon’s definitions, information can be measured by comparing the 
group of situations in which a cell (or a set of cells) spikes to the group of situations 
in which the cell (or set of cells) is silent. 4 Remember that Shannon defined informa¬ 
tion as separations between groups: spiking or not spiking is a yes/no question about 
something and carries information about that something, whatever it is. To answer how 
neuroscientists learn what that something is, we will have to wait until Appendix B, 
where we will turn to issues of measuring what cells are tuned to. 

How do cells get information to the next cell? This brings us to the concept of a syn¬ 
apse, which connects one cell to another. A spike in one cell releases a chemical (called a 
neurotransmitter ) across the very small (20 nanometers or 20 billionths of a meter) gap 
between the cells and changes the voltage of the other cell. Some synapses are excitatory 
and make the second cell more likely to fire spikes, while other synapses are inhibitory 
and make the second cell less likely to fire spikes. 


ions through, sodium channels, which only allow sodium through, calcium channels, chloride chan¬ 
nels, etc. How ion channels work and how the membrane voltage is generated is well understood, and 
descriptions can be found in many neuroscience textbooks. For those interested. Neuroscience, edited 
by Dale Purves and colleagues, is an excellent starting point. For those who want a more in-depth 
understanding of the channels themselves, Bertil Hille s Ion Channels of Excitable Membranes remains 
the classic text and is readable by an advanced undergraduate. 
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Of course, whenever were talking about biological systems, things are never so sim¬ 
ple. Actually, there are lots of different neurotransmitters and lots of different synaptic 
receptors to detect them. There are synaptic receptors on the presynaptic neuron. There 
are also synaptic receptors, called G-protein-coupled receptors, that, instead of allowing 
ions to flow in, release an attached protein complex, which then interacts with other 
proteins inside the cell. This engenders an extremely complex interflow of intracellular 
information processing. In general, we won’t need to go into detail about protein cas¬ 
cades for this book, but they are an important part of cellular processing, particularly in 
how cells change during learning. 

To a first approximation, neurons take information in through their dendrites, per¬ 
form computation on it in the dendrites and the soma, and then send information out 
through the axon, primarily by deciding when and whether to fire an action potential 
(spike), which travels down the axon and releases a chemical across a synapse connect¬ 
ing to other neurons in the brain. As with all first approximations, this description is 
often wrong in detail, even if correct in general. Some neurons are also performing cal¬ 
culations in their dendrites; some neurons connect through dendrite-to-dendrite syn¬ 
apses; some neurons do not fire action potentials; some neurons connect up to other 
neurons with electrical connections rather than chemical synapses; etc. But the starting 
description of a neuron with a dendrite, a soma, and an axon, connected to other neu¬ 
rons via synapses, is good enough for now. 

You can imagine each neuron poised between excitatory inputs encouraging it to 
fire and inhibitory inputs suppressing it. The brain spends its time poised in a dynamic 
balance between choices, like a baseball player up on the balls of his feet, ready to jump 
forward to steal second base or ready to dive back to first. It does this for exactly the 
same reason that the base runner does: so it can react quickly. 

Current theories suggest that many brain dysfunctions are conditions in which 
something has fallen out of balance. Epilepsy, for example, is almost certainly a conse¬ 
quence of an instability in excitation (encouraging neurons to fire spikes) and inhibition 
(holding them back). 5 What is interesting about the nature of dynamic balance (par¬ 
ticularly because of its dependence on feedback) is that you can get the wildfire spiking 
of epilepsy with either an instability in excitation or an instability in inhibition. Other 
diseases as well, such as schizophrenia, are also now being investigated as dynamic bal¬ 
ances gone unstable. 6 

This balance exists at all levels, from the systems level, where multiple decision-mak¬ 
ing systems compete for a decision, down to the neural level, where both excitatory and 
inhibitory inputs balance to keep neurons firing at just the right rate, to the subcellular 
level, where learning and memory reside. 


Memory 

Memory can be defined as changes that reflect a history so that future responses differ 
because of it. Just as anything that allowed us to differentiate two signals is information, 
anything that reliably changes in response to a past can be thought of as a memory of 
that past. The dent in the car fender “remembers” that it was hit. The fender will be 
weaker and may respond differently the next time. One of the tenets of this book is 
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that all of the psychological and mental observations that we observe have physical 
instantiations. 

The physical instantiation of memory resides in changes in the synapses that connect 
neurons one to the other and in the internal protein cascades within each neuron. The 
primary changes in the nervous system during learning occur in the synapses—how 
much of an effect the presynaptic cell has on the postsynaptic cell when the presynap- 
tic cell fires its spike. These changes are now known to exist throughout the brain, in 
the hippocampus, cortex, cerebellum, basal ganglia, and even the spinal cord. 7 We now 
know that these connection strengths depend on the internal milieu of the protein cas¬ 
cades within the cell and the timing of the presynaptic and postsynaptic spikes. 8 


A journey from sensory to motor 

It might be useful to take a journey from a sensory receptor into the brain. As we saw 
in our thermostat example at the beginning of the book, a decision process requires 
measuring the world, processing the information hidden within those measurements, 
and then acting upon that information. How does the neural system perceive the world? 
The first step is that there must be sensory neurons—neurons that fire action potentials 
in response to sensory stimuli. F Some cells, like the rods and cones in the retina, react 
through protein cascades. Rods and cones contain a light-sensitive protein called an 
opsin; in response to photons hitting the retina, these molecules change their shape, 
which causes a cascade of other protein changes, the last of which is to open up an ion 
channel and change the voltage in the cell. In other cells, the translation protein is a 
direct ion channel itself. For example, hair cells in the ear contain ion channels with 
little springs made of protein in them. As the sound wave moves the hairs on the cell, 
the hairs pull on the protein springs, stretching the ion channel open and allowing ions 
to flow in, changing the voltage of the cell. So, literally, these cells have sensors that 
translate or transduce the physical reality of the world into the inner information-world 
of our brains. 

On the other end, motor neurons send axons to muscles. Muscles are excitable cells 
just like neurons and can also fire action potentials (spikes). Because the muscles all 
line up in arranged structures, the firing of many action potentials in muscle tissue can 
be detected electrically quite easily. These muscle-based action potentials are what is 
being detecting in an electrocardiogram (EICG [the IC comes from the German Kardio, 
meaning “heart”]). In other muscle groups, these same signals are generally detected 
with electromyography (EMG [myo comes from the Greek word for muscle]). As you 
move any muscle in your body, the muscles are firing action potentials. The action 
potentials open up calcium channels in the muscle cell, which are then translated 
back into physical force by a calcium-sensitive protein ( myosin ) that contracts in the 
presence of calcium. 


p Actually, most direct sensory cells in the mammal do not fire action potentials. They release synap¬ 
tic chemicals at a rate proportional to their voltage. The next cells downstream (e.g., the visual ganglion 
cells in the retina or the neurons in the auditory nerve) do, however, generally fire action potentials. 
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The neural network that is the brain can take information in from the world through 
its sensory systems and can act on the world through its motor systems. This book is 
about the steps in the middle, how a network of cells, in constant dynamic flux, pro¬ 
cesses the information therein to take the right actions in the right situations. 

Books and papers for further reading 

• Eric Kandel (2006). In Search of Memory: The Emergence of a New Science of Mind. 
New York: Norton. 

• Joseph E. LeDoux (2002). The Synaptic Self. London, UK: Penguin. 

• Peter Dayan and Larry F. Abbott (2001). Theoretical Neuroscience. Cambridge, MA: 
MIT Press. 

The details of how neurons themselves work are given in the first few chapters of many 
neuroscience textbooks, many of which are actually easy to read for those who are not 
scared to dive in. 

• Dale Purves et al. (2008). Neuroscience. Sunderland, MA: Sinauer Associates. 

• Neil R. Carlson (2010). Physiology of Behavior. Boston: Pearson. 

• Eric Kandel, James H. Schwartz, and Thomas M. Jessell (2000). Principles of Neural 
Science. New York: Elsevier. 


Appendix B 

GLEANING INFORMATION FROM THE BRAIN 


Clouds hide the summit, 
but I create it in my mind. 
Seeing the mountain, 

I create it in my mind. 


Neurons transmit information by firing action potentials and connect 
to other neurons via synapses. As we have just seen, an action potential 
is a short change in the membrane voltage of the cell (usually a millisec¬ 
ond or so in duration). If we had a wire or other conducting medium 
near that neuron when it fired its action potential, it would be detectable 
on the wire as a quick spike. If we had a sensitive enough instrument, we 
could detect that spike. There are many technologies that allow us to see 
signals that reflect this activity. These technologies allow us to (in effect) 
read the animal’s mind, to ask what it is thinking about, to see what it 
sees, and to hear what it hears. We will also see technologies that allow 
us to manipulate (at a limited level) those physical signals and change 
what is in the animal’s mind, what it sees and what it hears. 


Representing information with spiking cells: Tuning curves 

Each neuron fires its spikes carrying information about some aspect of the world, some 
sensory signal, some motor action to be taken, some internal process, or some combina¬ 
tion of all three. One of the key questions in behavioral neuroscience is What informa¬ 
tion does a neuron's spiking pattern encode? 

We generally answer this question by relating the neural firing of a cell to some 
behavioral variable through a tuning curve. Classically, a tuning curve defines the 
likelihood that you will see spikes from a given neuron when an event occurs. For 
example, an auditory cell in the primary auditory cortex may be tuned to a specific 
frequency of sound. If you hear a tone at that frequency, that cell will fire spikes. 
Other cells will be tuned to aspects of that sound, the timbre, even the specific song 
if you recognize it. A visual cell in the primary visual cortex may be tuned to a spot 
of light, but other cells in deeper visual cortices (sometimes called “higher-order 
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cortices” because their representations require more processing) are tuned to faces, 
trees, or other objects. 

It is important to note that a cell’s tuning curve does not explain why the cell is 
tuned to that information in the world, only that the cell’s firing is correlated with it. 
Visual cells are tuned to visual signals because light impacts the retina, which fires the 
retinal ganglion cells, which transmits information to the thalamus, which then sends 
signals to the visual cortex, making the spikes fired by the neuron in visual cortex 
correlated to the visual input. Primary motor cortex cells are tuned to specific muscle 
movements because they send their output to motor-controlling neurons in the spi¬ 
nal cord, so their firing influences the muscle and their spikes are correlated with its 
movements. 

Many neuroscientists will say that tuning curves can only be applied to behavioral 
variables that directly relate to the immediate world of the animal, sensory signals 
(following a flash of light, a specific sound) or motor actions (preceding the movement 
of a specific muscle), but the brain is a physical object. When you think of your grand¬ 
mother, some cells are firing that represent the information about your grandmother. 
If you think about what she looked like the last time you saw her, cells are firing that 
represent information about her face. If you think about her voice, cells are firing that 
represent information about what she sounded like. 

Much of the older popular neuroscience literature talked about the “grandmother 
cell,” implying that there was a single cell in the brain that fired to thoughts of your grand¬ 
mother and only to your grandmother. As we will see later, this hypothesis is incorrect. 
Representations in the brain are distributed. Some cells will fire more to women than to 
men, other cells to old people more than young, and still other cells to specific aspects 
of your grandmother such as her voice or her face, and it is that set of cells that represents 
the concept of your grandmother, not any one of those cells. 

To understand the concept of a distributed representation, imagine turning on the 
television to find it showing a football game. You want to know which teams are playing 
because you want to decide if it’s a game you want to watch, but the camera is just pass¬ 
ing over the audience. (To make this thought-experiment work, you’ll need to ignore 
the fact that modern television sports programs have little text-boxes in the corner of 
the screen that tell you who’s playing and what the score is.) From the clothes of the 
fans, you could probably make a very good guess about the two teams. Looking at any 
one fan won’t tell you. Maybe that person just happens to like that team. Fans might not 
be wearing the team jersey, but they’ll likely be wearing clothes with the team colors. If 
you notice that most of the fans are wearing a specific team’s colors, that’s a pretty good 
indication of which teams are playing. You’ll probably notice that there are two teams’ 
jerseys and colors that are much more commonly represented. The fans are representing 
information about which teams are playing. In fact, you can probably determine which 
team is doing well at a given moment by the expressions on the faces of one team’s fans 
versus another. The fans are a distributed representation of the two teams—you can 
learn a lot about the game just by watching the fans. 

Mathematically, a cell can represent any variable as long as its spiking is reliably 
related to that variable. For example, cells in the rodent hippocampus are tuned to 
the location of the animal. 1 As the animal runs around an environment, a given place 
cell will fire spikes in a preferred location (the place field of the cell). This is not a 
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sensory response, nor is it a motor initiation signal; it is actually a representation of 
location. A 

Tuning curves often entail firing strongly to a single preferred stimulus, which then 
falls off as the stimulus differs from that preferred stimulus. For example, auditory neu¬ 
rons in the primary auditory cortex have a preferred frequency at which they fire the 
most spikes; as the supplied sound frequency drops off in each direction, the number 
of spikes that the cell fires decreases. 2 Each auditory neuron has a different preferred 
frequency. When you hear a note, each cell will fire spikes reflecting the harmonics of 
that note. Place cells in the hippocampus have a preferred location. At the center of 
that place field, the cell fires the most spikes the most reliably; at distances from that 
place field, the cell fires fewer and fewer spikes, until it becomes quiet. Each place cell is 
tuned to a different location in space in a different environment. The representation of 
a position in the environment entails a distribution of cells firing, some (with preferred 
locations near the position) firing a lot and some (with preferred locations far from the 
position) firing only a little. 

However, tuning curves don’t have to have a single preferred stimulus. 3 All that mat¬ 
ters is that the spiking of the cell be reliably informative about the behavioral variable. 
Because information is about defining groups, this is equivalent to saying that the behav¬ 
ioral variable must be reliably informative about the spiking times of the cell. 4 

This means that we can determine information about the animal’s representation 
of the world from the firing of cells. Let’s go back to place cells. Each place cell is tuned 
to a different location in space. This means that we can invert the tuning curve—if 
we know what location is represented by that place cell, then when that place cell 
fires spikes, we can infer that the animal is representing information about that loca¬ 
tion. Even so, a typical place field is pretty large, (in rats, place fields can range in size 
from 20 centimeters to meters in length. 5 ) Even more importantly, the relationship 
between firing spikes and the location of the animal is not 100%. Sometimes a cell 
fires spikes when the animal is outside the cell’s place field; sometimes the cell doesn’t 
fire when the animal is in the place field. But if we have two cells with overlapping 
place fields, we can be more accurate because we can determine if the first cell is fir¬ 
ing but not the second, if the second cell is firing but not the first, or if both are firing. 
Current technologies allow us to record as many as several hundred cells, simultane¬ 
ously, from awake, behaving animals. By integrating information from hundreds of 
cells, we can decode location quite accurately. 6 From several hundred cells, we can 
typically decode location to an accuracy of better than 1 centimeter! We can even 
determine when the cells are representing a location different from the animal’s cur¬ 
rent location; 7 we can determine where the animal is imagining or thinking about. 
We can read the rat’s mind. 


A How these place cells determine the location of the animal in order to decide when to fire spikes 
is a complex issue—the rat does not have a built-in GPS to determine its location. What actually hap¬ 
pens is that it keeps track of its position by dead reckoning—if I know where I am and I take a step 
forward, I still know where I am, I’m one step forward of where I was—and learns to associate spatial 
cues (like landmarks) with those location representations, allowing the animal to reset its representa¬ 
tion when lost. For those interested, my earlier book Beyond the Cognitive Map describes this process 
in great detail. 
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Other signals 

Local field potentials (LFP) 

Since we’ve already talked about recording electrical signals and identifying neural fir¬ 
ing patterns from them, a good next signal to discuss is the local field potential ( LFP ). 
If one takes the same electrical signal appearing on the electrodes and listens to the 
low-frequency components instead of the high-frequency components, one can record 
LFPs. B Currently, no one is completely sure how LFPs are generated, but they seem to 
be related to the slow electrical currents arising from synapses. 8 

In many brain structures like the hippocampus and the basal ganglia, nearby cells 
do not tend to be processing the same information, and LFP signals do not carry 
information about the task at hand, but they carry information about how the brain is 
processing those tasks. 9 These LFPs tend to indicate the state of the system and most 
likely reflect different processing regimes. For example, when you are attentive, actively 
listening to someone talking to you or actively trying to understand new information 
you are reading about, the LFP in your hippocampus shows a 7 to 10 Hz, almost sinu¬ 
soidal rhythm, called “rhythmic slow activity” (RSA) or theta (for the frequency band 
it covers). In contrast, if you are fading out, not really listening, losing your attention, 
nodding off, the LFP in your hippocampus shows a more broad-spectrum arrhythmic 
activity called “large amplitude irregular activity” (LIA). 10 

In contrast, in the mammalian neocortex, nearby cells do process similar informa¬ 
tion. 11 (One of the most remarkable things about the mammalian cortex is that although 
the brain itself is unquestionably three-dimensional in its connectivity and its structure, 
the cortex itself is more of a sheet than a three-dimensional structure. The six layers 
of cortex form a thin sheet that changes tremendously in area but not much in depth 
between mammals. Evolution seems to be taking a specific template and copying it in 
two dimensions to make larger and larger cortices. 12 ) Because the processing units in 
cortex are arranged in small 0.3-millimeter-wide columns, where all the cells in a col¬ 
umn are processing similar things, there is evidence that LFPs in the cortex can carry 
signals about the information in that column. 13 

Imagine you are trying to understand the game of American football but you can’t see 
the game. All you have is a microphone, which you could lower to listen to a single person. 
Perhaps you get to listen to the quarterback or the coach and learn a lot about the game. 
This is like single-cell recording, where you find that the signal on your microphone tells 
you a lot about the behavioral variable (the game). On the other hand, you might end up lis¬ 
tening to the drunk fan in the third row, or the businessman discussing stock sales, and not 
find much relationship to what is going on in the game. If, however, you pull your micro¬ 
phone up and listen to the roar of the crowd, you can also learn about the game. If you hear 


B Remember that spikes are 1-millisecond blips, so they are best seen with frequency filters set 
around 1 kilohertz (changes happening on the order of 1 millisecond). Spikes are usually recorded 
using filters set from 600 Hz to 10 kHz, which pick up things happening on the order of a millisecond 
or faster. LFPs are usually recorded using filters set from 0.1 to 400 Hz, which pick up things happening 
on the order of tenths or hundredths of a second. 
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the crowd chanting “Defense! Defense! ” or the crowd being really quiet, you can determine 
pretty accurately which team is on offense when. You can learn when a really good (or really 
bad) play happens just by listening to the volume of the crowd. That’s like LFP in the hip¬ 
pocampus or the basal ganglia. If all the fans from one team are sitting together and all the 
fans from the other team are sitting together, you might even be able to tell which team is 
winning by the volume of the shouts of each group. That’s like LFP in the cortex. 


The electroencephalogram (EEG) 

Many people are familiar with a form of LFP recorded outside the skull: the electroen¬ 
cephalogram (EEG). 14 EEG electrodes are placed around the scalp and record the same 
sorts of waves as LFPs. Just as with LFPs, EEGs are primarily used to categorize brain 
states. Many people, for example, are familiar with slow-wave and REM sleep, which can 
be identified by EEG oscillations. Different moods and attention states can also be mea¬ 
sured by EEG changes. For example, calm, relaxed states are often identified with EEG 
rhythms (arising from the cortex) in the 8 to 12 FIz range (confusingly called “alpha 
waves” in the EEG literature and “theta waves” in the LFP literature). 

People can learn to control their EEG rhythms. 15 There are now games available 
that use EEG signals to control balls or lights. Touted as allowing one to “control the 
ball with your mind,” these games are measuring EEG signals from a headset and using 
the ratio of the power in different frequency regimes (usually alpha [8-12 Hz] to beta 
[12-30 Hz]) to change the speed of a fan that blows a ball up or down. 16 

When EEG signals are measured in response to impulse events rather than looking 
at the ongoing frequencies in the oscillation, these signals are sometimes referred to as 
event-related potentials (ERPs). 17 They occur when a single event drives enough activity 
in the brain to be detected as a sharp change in the EEG. 

One of the most common ERP signals studied is the error-related negativity (ERN), 
in which EEG recorded from the frontal part of the scalp changes in response to making 
an error. 18 Interestingly, this activity occurs approximately 100 milliseconds after one 
starts the movement, which can be long before the movement is complete. (The ERN 
seems to reflect that “oops, that’s wrong” signal we sometimes feel when we react too 
quickly.) We now know that the ERN arises from signals in the anterior cingulate cor¬ 
tex, which is involved in monitoring actions and allowing one to take a different action 
than one usually might. (See, for example, the discussion of overcoming a phobia and 
facing one’s fears in our discussion of self-control in Chapter 15.) 


Recording spikes from human patients 

The primary information-measuring experiments in this book come from animal 
experiments because we have the technology to record from single neurons and neu¬ 
ronal ensembles in animals. c Such technology works in humans as well but requires 


c It is important to note the great care taken by neuroscientists, particularly behavioral neurosci¬ 
entists, in the treatment of their animals. A distressed animal does not perform tasks well, particu¬ 
larly complicated tasks like those required for decision-making experiments. All animal experiments 

( continued ) 
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implantation of wire electrodes into the subject; therefore, this is done only for clinical 
treatment. 

Sometimes scientists are allowed to piggyback onto the clinical implantation and 
(with the permission of the subject and the oversight of a review board) record from 
electrodes implanted for other reasons. Two examples where such electrodes are rou¬ 
tinely implanted are for the determination of the initiation sites of epileptic seizures and 
for deep-brain stimulation for the treatment of Parkinson’s disease. 

Briefly epilepsy entails neurons going haywire. The first treatment attempted is gen¬ 
erally antiseizure drugs, but that works for only a limited portion of the epileptic popu¬ 
lation. 19 If pharmacological treatments don’t work, an alternate option is to literally burn 
away the part of the brain going haywire. Of course, damaging the wrong part of the 
brain is extremely problematic (we have seen some tragic examples in this book), and 
physicians are very careful to do their best to ensure that they get only the right starting 
point for the epilepsy and that the part they are going to remove is not involved in criti¬ 
cal intellectual faculties. To do this, they implant electrodes and record single neurons 
and LFPs as the subject does tasks. 20 Since having the electrodes in one’s brain is not a 
painful process and recording from neurons does not damage the brain, many patients 
are happy to do a few extra cognitive tasks and to allow scientists to record from their 
brain during the procedure. 

These recordings, however, can be taken in only very limited situations (during the 
rare times when a patient has a clinical reason for having electrodes implanted) and in 
patients with dysfunctional brains (that’s why they’re getting the clinical treatment). To 
study normal function in nonpatients, we need a technology capable of recording neural 
signals from awake subjects. We’ve already discussed EEG. There is a related measure¬ 
ment technology called MEG that works like EEG but uses the magnetic component of 
the electromagnetic wave. Although MEG is less distorted by the skull than EEG, both 
MEG and EEG suffer from an inability to localize the signal. These technologies can tar¬ 
get “frontal cortex” versus “visual cortex” (which is in the far back of the brain), but they 
can’t tell you what is going on in the shape-recognition component of the visual cortex, 
or the striatum of the basal ganglia, or the hippocampus. 


Functional magnetic resonance imaging (fMRl) 

If you’ve been following neuroscience over the past few decades in the popular press, 
you are probably wondering why I haven’t brought in functional magnetic resonance 
imaging (fMRl) until now. fMRI has become beloved by the popular press because it 
allows insight into the brains of normal human subjects. It creates beautiful pictures. 
It can be run on normal people performing complex, human tasks, such as language 
and cognition. But what’s really important to understand about fMRl (and its cousins, 
such as positron emission tomography [PET]) is that it measures blood flow, not neural 
activity. 21 


done in any modern university are required to be approved by an animal-care committee that includes 
both veterinarians and community activists as well as scientists. This committee is tasked not only 
with checking the treatment of the animals but also with determining if the experiment being done is 
“important enough” to justify the use of the animals. 



Appendix B 


253 


fMRI measures the magnetic spin of hemoglobin molecules in the blood. 
Mammalian blood carries oxygen to the tissues by binding it to the hemoglobin mol¬ 
ecule. Hemoglobin with bound oxygen is redder than unbound hemoglobin; it also has 
a different resonance to magnetic changes. An fMRI machine is a very big, very strong 
set of magnets. By changing the magnetic field in specific ways, it can detect the level of 
oxygen in the blood. 

Modern high-field magnet fMRI machines can measure the oxygenation level in 
approximately a one-millimeter cube of tissue. While a 1 mmXl mmXl mm voxel 
(volume-pixel) sounds small, individual neurons are typically 50 micrometers in diam¬ 
eter (8000 times smaller). For computational reasons, fMRI experiments often use 
much larger voxels, even as large as 3 or 4 mm on a side. (PET requires an even larger 
voxel size.) That means that each voxel could contain a hundred thousand neurons. 
Even allowing that neurons are only a small portion of the brain tissue (most of which 
contains support structures, like capillaries, glial support cells, and the axons and den¬ 
drites of the neurons), that voxel still contains tens of thousands of neurons. 

So what does blood oxygenation have to do with neural activity? At this point, 
we know they are related, but not exactly how. This is still an area of active investiga¬ 
tion in neuroscience. 22 We know that when an area is “neurally active” (for example, 
your primary visual cortex is more active when you are looking at a flashing visual 
scene, like a movie, than when you are closing your eyes and listening to an auditory 
scene, say listening to music), the cells are using glucose, drawing extra energy from 
the body. 23:D The overworked neurons send a signal to the capillaries to allow more 
blood to flow. The neurons don’t actually need more oxygen, so there’s more blood 
flowing, but the same amount of oxygen is pulled out, so what’s flowing by becomes 
less deoxygenated, which is detectable with fMRI. These changes in blood flow 
are called the hemodynamic response, which occurs over the course of about 4 to 
5 seconds. (Remember that a single neural spike is 1 millisecond—that is, l/1000th 
of a second!) Typical human reaction time to a surprising stimulus is only a few hun¬ 
dred milliseconds.® fMRI researchers have developed ways to infer the “neural activ¬ 
ity” from the hemodynamic response to identify the timing, by essentially relating 
the blood flow to events that occurred 4 to 5 seconds before the signal was detected, 
but even that can’t get to what single neurons are doing or the specific timing of 
their spikes. 

It’s not clear at this point what is causing the neurons to spend all that energy. 25 This 
means that fMRI can tell you which parts of the brain are working hard, but it can’t tell 
you how the neurons in that part of the brain are processing information. 


D The brain is a very energy-costly component of your body. Even though your brain is only a cou¬ 
ple of percent of your body weight (a typical brain weighs a little less than 1.5 kg [3 lbs]), it consumes 
about 20% of your typical energy reserves. 24 

E This is the key to baseball—a 100-mph pitch takes 400 milliseconds to reach the plate. This means 
that a batter has about less than half the distance between the mound and the plate to decide whether 
or not to hit the ball and to determine where the ball will be and when it will cross the plate. Of course, 
if a batter can predict what pitch is going to come before the pitcher throws the pitch, the batter can 
be better prepared to recognize the pitch. This is the essence of intelligence—using information and 
expectation to help make better decisions quickly. 
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Correlation and causation 

All of the technologies described above are correlational. They ask to what extent two 
signals change together: a neural signal (firing of neurons, LFPs, EEC, blood flow in the 
brain) and a behavioral variable (the location of the animal, a sensory signal, a motor 
action). Logically, correlation is not causation. Just because two things co-occur does 
not mean that one has caused the other. Correlation is necessary (though not sufficient) 
for causation, and, in practice, it’s a pretty good indicator of causation. Nevertheless, 
scientists in general, and medical scientists in particular, are particularly concerned with 
causation because they do not want to make the mistake of thinking that two things that 
co-occur are causing one another. They have, thus, developed several technologies that 
allow control of the system. 


Lesions 

The oldest method of manipulating the system was to damage it physically, to lesion 
part of it. If that part is critical for accomplishing some task, then the animal (human or 
otherwise) will be impaired at that task. Obviously, one cannot go in and lesion human 
tissues for experimental purposes. However, the brain is a particularly sensitive biologi¬ 
cal organ, requiring constant blood flow. Cutting off the blood (or even just the oxygen) 
supply for even a few minutes can lead to neuronal death, leaving the person with an 
unfortunate brain lesion. 26 This is what happens during a stroke, for example. 15 

Because of the way that the blood vessels in the brain branch, a common stroke 
occurs above the right parietal cortex (just above and behind your right ear). 28 When 
this part of the brain is damaged, people have trouble recognizing the left side of visual 
objects. 0 This inability to recognize objects on one side of the visual field is called hemis- 
patial neglect because it entails the inability to process (it neglects) half of (hemi) the 
spatial visual field. Patients with hemispatial neglect will draw only the right side of a 
clock, and will see the clock at all only if it is on the right side of their visual field. 

Hemispatial neglect even occurs in imagined spaces. In a classic experiment, Eduardo 
Bisiach and Claudio Luzatti asked hemispatial neglect patients to imagine the Piazza 
Del Duomo in Milan (the patients’ native city). 30 They asked the patients to imagine 
standing on the south end of the Piazza and to name the buildings in the square. The 
patients each listed only the buildings on the east side (their right). When they were 
asked to imagine standing on the north end and to name the buildings, they named 
only those on the west side (their right). When pressed, they said that was all they could 


7 Actually, in a stroke, several damaging processes occur over the course of minutes to hours to days. 
One of those processes is that the decreased blood flow can mean that cells fall out of balance and start 
to fire at high rates, which drives the cells to death. Treatment that can reach the cells before they die 
can help reduce or even prevent brain damage due to stroke. 27 

G Sensory and motor systems cross in the brain. Thus, the left visual field and the left hand are both 
processed by the right side of the brain. What, in the evolution of the systems, drove them to cross is 
still being debated, but there is some evidence that early prey animals have direct connections from 
sensory systems to the opposite-side muscles. 29 If something is coming at you on the left side, you want 
to go right fast! 
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imagine, even though they had named the other buildings only a few minutes earlier! 
Visual imagination uses the same perceptual systems that we use in seeing. 31 

Other common strokes notoriously affect certain motor and language areas, leading 
to inabilities to process language. Because different strokes affect the ability to recog¬ 
nize or to produce language, even as early as the 1870s, Paul Broca (1824-1880) and 
Carl Wernicke (1848-1905) were able to separately identify the locations of the human 
brain that controlled speech production (now called Brocas area ) or speech recognition 
(now called Wernicke’s area). 32 

Brain lesions can also arise from medical procedures, some intentional, others not. For 
example, epilepsy that cannot be controlled by pharmacological means (with drugs) can still 
sometimes be controlled by taking out the focus or starting site. 33 Epilepsy is like an ava¬ 
lanche or a wildfire: it has a starting point, from which it spreads. If that starting point is 
consistent from seizure to seizure within a patient and can be found, it can be removed. Of 
course, removing a part of the brain damages its ability to process information, but some¬ 
times a balance has to be found between living with epilepsy and living with a damaged brain. 
Hippocampal resection (removal) in epileptic patients (such as H.M.) was critical to the dis¬ 
covery that there are at least two separate decision-making systems (Deliberative, dependent 
on hippocampal function, and Procedural, independent of hippocampal function). 34 

The problem with inferring function from lesion studies is that solving tasks requires 
lots of components. Let’s go back to the thermostat analogy in Chapter 2—there are 
lots of lesions that could be made to the thermostat that would make your house cold. 
The furnace could be broken. If it’s a gas furnace, the pilot light could be out or the gas 
turned off. The temperature sensor could be broken. All of these would cause the fur¬ 
nace not to be on and the house to be cold, but we wouldn’t want to conclude that the 
temperature sensor and the pilot light were doing the same thing. 

The other problem with lesions is that of compensation. As biological organisms, we 
have backup systems. (During most of evolution, there was no repair shop to take our¬ 
selves to if something broke down, so animals that evolved backup systems were more 
likely to survive and procreate than those without.) If an animal has had enough time to 
recover from the lesion, it may have found new ways to do something and may be able to 
solve the same tasks in different ways from how an intact animal would solve them. 


Temporary inactivation of parts of the brain 

To prevent compensation, and since one cannot morally control the lesions one sees in 
humans (the only human lesions one is going to see are caused by natural phenomena), 
scientists have developed several complex techniques to inactivate parts of the brain 
reliably. The most common technique used in animals is pharmacological, in which 
a very small amount of chemical that temporarily affects the neurons is infused into 
an area of the brain. For example, lidocaine blocks a certain kind of sodium ion chan¬ 
nel, which prevents neurons from firing. When the dentist injects lidocaine under the 
gum, it prevents the nerves in the skin from communicating with the brain, leaving your 
mouth numb to any pain that would be caused by the subsequent procedure. 

In humans, one does not want to inject chemicals into the brains of experimental 
subjects, but a new technology has been recently developed called transcranial magnetic 
stimulation (TMS). 35 TMS creates a quickly changing magnetic field that can stimulate 
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and then disrupt neural firing in a targeted area of cortex. The effect is disruptive but 
transient, wearing off after a few minutes. H 


Genetic manipulations 

Because the neuronal processes (such as the synaptic channels that listen to specific 
neurotransmitters) depend on the genetic structure of the cell on an ongoing basis, 
changing that genetic structure changes how the cell processes information. New tech¬ 
nologies are now available that allow direct manipulation of that genetic structure. 37 ; * 1 
If the DNA is changed (say to include the information to build a new, different kind of 
ion channel), then the cell will follow that DNA. 

We now know that DNA is more like a programming language, with “if-then” state¬ 
ments and other sorts of local control (as in, “if protein X is present, then also generate 
protein Y”). This means that modern genetic manipulations can express new proteins 
(new ion channels, new processes) so that they appear only in a limited subset of cells, 
such as only in the CA3 neurons of the hippocampus; or so that they appear only in the 
presence of an extrinsic chemical, such as the antibiotic doxycycline; or so that they 
only appear after a certain time, such as only in cells that are active (firing a lot of spikes) 
during the task. 39 These manipulations produce changes in the abilities of animals to 
solve tasks, which informs us how the system works, physically. 

In the past few years, researchers have developed a new technology where a light- 
sensitive ion channel is genetically implanted into an animal. 40 These channels produce 
voltage changes in response to certain frequencies of light. This means that if a light is 
shined on cells with these channels in them, the cells can be induced to fire spikes or 
prevented from firing spikes. Because these ion channels can be placed in specific cells 
in specific brain areas, it can give remarkable control of the neural mechanism. This is 
a burgeoning field called optogenetics and has the potential to revolutionize our under¬ 
standing of the actual mechanisms of brain function. 


Stimulation—electrical manipulations 

Of course, cellular systems are also electrical, and one can also pass current into the 
system to manipulate it. For example, a common treatment for Parkinson’s disease is 
deep-brain stimulation, in which an electrode is placed into a structure (commonly the 
subthalamic nucleus, a part of the basal ganglia). Regular stimulation from this electrode 
(at between 50 and 200 Hz, depending on what works for the patient) somehow resets 
the subthalamic nucleus, allowing it to process information correctly again. 41 Although 
deep-brain stimulation works wonders in some patients, we don’t know why, we don’t 


H The safety of TMS is still being debated. 36 For most people, it seems that the short-term effects 
wear off pretty quickly, but the long-term effects are still unknown. 

1 It should be noted that we have been able to manipulate genes for thousands of years through 
selective breeding. 38 Humans bred dogs to be more friendly, cows to give more milk, and com to be 
edible long before modern genetic manipulations. Modern technologies just make the manipulation 
more direct. 
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know what the mechanism is. There are lots of theories, including that it prevents neural 
firing, thus shutting down a dysfunctional structure; that it resets neural oscillations 
that have become pathological; and that it is making it easier (or harder) for axons that 
pass by the structure to transmit information. 42 

Stimulation can also be used to manipulate information being represented within 
the brain.- 1 In the mid-1990s, William Newsome and his colleagues were able to change 
decisions made by a monkey watching a visual stimulus by stimulating certain parts 
of the visual cortex. The monkey had to decide if a set of dots on a screen was mov¬ 
ing in one direction or another. By stimulating the part of the brain that represented 
visual motion, they could change the monkey s decision, presumably by changing its 
perception. 43 

Actually, stimulation to write information into the brain has been used in humans 
for over 40 years. Cochlear implants work by laying electrical contacts along the inner 
ear. 44 An external microphone translates the sound into the appropriate electrical signal 
that the ear cells expect. The electrical contacts stimulate the cells, allowing the brain to 
hear sounds. 

Several laboratories have attempted to use retinal implants to restore vision in blind 
patients. 45 Similar to the cochlear implants, external images are translated by a camera 
to electrical signals that are sent to stimulation electrodes in the retina. Early clinical tri¬ 
als are now under way. One of the major complications of both the cochlear and visual 
implants is that the stimulation has to match the neural representations that the brain 
expects. By understanding the principles of representation, we’ve been able to begin the 
process of cybernetics, actually recovering lost function. 

Books and papers for further reading 

• Andrea M. Green andjohn F. Kalaska (2011). Learning to move machines with the 
mind. Trends in Neurosciences, 34, 61-75. 

• Adam Johnson, Andre A. Fenton, Cliff Kentros, and A. David Redish (2009). 
Looking for cognition in the structure in the noise. Trends in Cognitive Sciences, 13, 
55-64. 

• Frederick Rieke, David Warland, Rob de Ruyter van Steveninck, and William Bialek 
(1996). Spikes. Cambridge, MA: MIT Press. 


J If you still need convincing at this point that the mind exists within the brain, this is the final proof. 
Manipulating the spiking of cells within the brain changes the perception, the decision, and the action 
taken. We truly are physical beings. 
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Appendix C 

CONTENT-ADDRESSABLE MEMORY 


Cacti glisten in the sun. 

Then clouds hid the mountain, 
the streets like canyons rushed with water... 

I smell sweet 
prickly pear in bloom. 


Memories in the brain are stored as patterns of firing across 
cells and are retrieved by the content they contain. This process 
is called "pattern completion" because a memory is retrieved by 
completing a partial pattern of cellular firing. The neural mecha¬ 
nisms of this are well understood and have important implica¬ 
tions for how we recognize and remember, which has important 
implications for decision-making systems. 


When a digital camera takes a picture of your kid’s birthday party and writes it onto 
its flash memory card, it translates that picture into a set of on-and-off signals 
(“bits” A ), which it writes onto the card. It then writes the location into the “file allocation 
table,” which is basically an index of locations on the memory card. When you want to 
recall the picture, the camera checks the index, follows it, and reads the file. When the cam¬ 
era first stored the picture on its memory card, it gave it a name like img_3 541.jpg, 
which tells you nothing about the content of that image. Your brain can also store infor¬ 
mation like the memory of your kid’s birthday party in small changes across many neu¬ 
rons, in this case through changes in the connection strength between neurons. But 
there’s no central index. Instead, the full memory is retrieved from the partial content— 
the smell of the birthday cake, the sound of kids’ laughter, the sight of a specific toy. 


A Remember that one bit is the answer to oneyes/no question (Appendix A). Eight bits = one byte. 
When your computer has 4 GB of memory, that’s 4 gigabytes, or 4 billion bytes. Of course, it wasn’t 
that long ago that computers only had 1 MB (one megabyte = 1 million bytes) or even 640 I<B (one 
kilobyte = 1 thousand bytes). Now, the flash card in my digital camera has 32 GB, and even larger ones 
are available. 
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Retrieval 
by index 

img_1054.jpg 
Stonehenge.jpg 


img_2063.jpg 
himeji.jpg 


img_5370. jpg 
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Figure C. 1 Content-addressable Memory. Information can be addressed by indexes 
(whether by related names [such as Stonehenge . jpg] or by unrelated names [such as 
img_l 0 5 4 . j pg] ) or by content-addressable memory. Content-addressable memories 
are stored so that similar pictures and similar concepts are linked. Thus, a blurry image of 
Stonehenge, a paper-cutout image of Himeji, and a pencil drawing of Notre Dame retrieve 
their respective pictures. Similarly, memories can be retrieved by conceptual content. 


The term content-addressable memory comes from the computer science and psychol¬ 
ogy literatures of the 1980s. In computer science, the term was used to contrast with 
index-addressable memory, which is how your laptop, desktop, or smartphone stores 
information. 1 In psychology, the term was used because human memory is triggered by 
content, like the sights, sounds, or smells of the birthday party. 2 (Compare Figure C.l.) 
The concept, though, comes from Donald Flebb’s famous 1949 book. 3 


Cell assemblies 

One of the most famous sentences in neuroscience is “cells that fire together wire 
together,” which is supposedly from a 1949 book on neuropsychology by the psycholo¬ 
gist Donald Hebb. However, that sentence isn’t actually in Hebb’s book; instead, the 
citation is to a paragraph that concludes, “When an axon of cell A is near enough to 
excite a cell B and repeatedly or persistently takes part in firing it, some growth process 
or metabolic change takes place in one or both cells such that As efficiency as one of the 
cells firing B, is increased” (Hebb, 1949/2002, p. 62). From this paragraph, we get the 
terms Hebbian synapse and Hebbian learning. 

The concept that co-firing leads to an increase in connectivity almost certainly pre¬ 
dates Hebb. In a sense, it is the obvious translation of the theories that mentation is asso¬ 
ciative, which goes back to David Hume in the 1700s, if not earlier. 4 Hebb himself says 
that the general idea is an old one (Hebb, 1949/2002, p. 70). In his historical perspec¬ 
tive on LTP, Bruce McNaughton tells that Hebb was surprised at the excitement about 
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the “Hebbian synapse,” that it dates at least to Lorento de No, and that it is the only 
obvious mechanism of association. 5 In fact, the famous paragraph in Hebb’s book is the 
assumption step in his theory, not the prediction step: the full paragraph starts with “Let 
us assume that...” What Hebb was really proposing was that if we see this mechanism in 
a cortical system, then we would get cell assemblies and content-addressable memory. 

Hebb defined a cell assembly as a set of cells that worked together to represent a spe¬ 
cific set of information. Sometimes the term “cell assembly” is used to identify a single 
thing—thus, the set of cells active when you think of the dog you had growing up forms 
a cell assembly. Sometimes the term is used to identify a set of things—thus, the set of 
cells active whenever you see or think about dogs would form a cell assembly. 

The important concept in both cases is that it’s not a single cell that forms these sets. 6 
In both cases, we’re talking about groups of cells that work together. A single cell may 
participate in multiple cell assemblies, but some cells will differ between them. This 
means that it will only be in the aggregate that an observer can differentiate represen¬ 
tations in this population of cells. We can ask what the tuning curve is from a single 
cell, but we cannot decode the representation from it. For that we need a population of 
cells, an ensemble of cells. This kind of a representation is called distributed because the 
representation is distributed across the population. 

Hebb’s concept was that cells that represent, say, the sound of that dog barking, 
the color of its fur, and how you felt when you hugged it would all be linked together 
because these experiences tended to co-occur. 7 Thus, the taste of ratatouille made the 
way your mom made it could bring back a flood of memory—of your hometown, your 
youth, and that family emotion that can melt your heart. 8 The Hebbian synapses would 
complete a part of the cell assembly into the whole memory. The memory could be 
addressed by its content alone. 

Hebb’s conceptual derivation occurred before the necessary mathematical basis that 
was needed to study it, and the concepts of cell assembly and content-addressable mem¬ 
ory were not tested or explored until the 1980s and 1990s, after the development of the 
needed computational and neurophysiological technologies. 9 


Memory as pattern completion 

Computationally, content-addressable memory is equivalent to completing a partial 
pattern. 10 Imagine a degraded picture, where some of the pixels have been replaced with 
noise. With enough noise, so much of the picture will be changed that you won’t be able 
to recognize it anymore, but for a long time before that, you will still be able to recognize 
the picture, even through the noise. 

Computer models of this have shown that a very good system for completing par¬ 
tial patterns is to store connections between each pixel representing the similarities and 
differences between the pixels. For simplicity, let’s imagine a black and white picture, 
where each pixel is either on or off. If two pixels are both on, then they can support 
each other, but when the pair of pixels are opposite to each other (one on, the other 
off), they counteract each other. This is a network originally studied byjohn Hopfield, 
who showed in a very famous 1982 paper that such a network will complete a degraded 
picture to an original stored picture. 11 Hopfield was a physicist at CalTech at the time. 
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Using mathematics originally derived in physics to describe how iron becomes mag¬ 
netic when it cools down (the magnetic poles of the individual atoms slowly align), he 
showed how you can define an error or energy surface to understand how the system 
settled from the noisy, degraded picture into the remembered picture. 

Imagine that we represent the activity across all of our neurons as a point in a very 
high-dimensional space. 12 Each neuron provides a dimension and the position along 
that dimension is the firing rate of the neuron. This is easiest to think of with only two 
neurons, so we have only two dimensions, but the mathematics works with any number 
of dimensions. At any moment in time, our firing pattern across the population of neu¬ 
rons is a point in this n-dimensional space. We can describe the error (energy) function 
as height in a third dimension ((n + l)th dimension). This gives us a surface with hills 
and valleys. What Hopfield was able to show was that his network would always roll 
downhill and that the synaptic weights in his network could be constructed so that the 
bottom of the valleys correctly corresponded to the stored (remembered) patterns. The 
connection strength between the cells of the pattern determine the depth of the basin 
representing that pattern, and the likelihood that one will roll down into it. 13 

So imagine that we store some pattern (say a picture of Rodin’s The Thinker ) in our 
network (Figure C.2). This means that the pattern of activity across the network that 
corresponds to our picture of that iconic image is going to be at the bottom of one of 
these valleys. We then degrade that pattern by adding noise. When we put that degraded 
pattern into our network, it starts at some point different from our stored memory. But 
then the activity of our network starts to change—neurons that are off, but have lots of 
other connected neurons that say they should be on, start to turn on; neurons that are 
on, but don’t have a lot of support to stay on, turn off. As the activity pattern changes, we 
are moving along that energy/error surface. We are a ball rolling downhill. Eventually, 
we will arrive at the bottom of the valley of The Thinker. 

Of course, as we saw in our discussion of actual neurons (Appendix A), neurons are 
not off or on. Instead, they fire spikes, but these spikes are in response to signals arriving 
in the neuron’s dendrites, which depend on the firing of other neurons and the synaptic 
weight between those other neurons and the neuron in question. This produces a net¬ 
work of neurons working together. 



E=0 


Figure C.2 Degraded Patterns. We can measure the difference between patterns 
as the mean squared error E. As the pattern changes from the remembered pattern, the 
error increases. We can use this error as an energy function. If a means can be found 
to move down in this “error” function, we will find ourselves completing the pattern, 
“remembering” our image. Content-addressable memories work this way—by moving 
downhill of the error or energy function. 
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The concept of this network of neurons working together to recall a memory is 
sometimes called connectionism, because it is the connections (synapses) between the 
neurons where the memory is stored.® Hopfield’s contribution was to simplify the 
model to the point where he could define the energy or error function and prove that his 
system would recall the stored memory through pattern completion. In fact, these sorts 
of models work well, even if we only change the excitatory connections and use inhibi¬ 
tion only to maintain an overall level of activity, or if we have graded neurons, which are 
trying to recall firing rates, or if we have detailed models of spiking neurons. 15;C These 
models have been able to explain how changes in neuromodulators (like dopamine, ace¬ 
tylcholine, or serotonin) can change the ability of a network to store memories, as well 
as very specific (and often surprising) firing rate patterns observed in neural recordings 
from behaving animals. 22 What these models give us is an explanation for memory as 
categorization, which has a number of interesting consequences. 


Memory as categorization 

What if you have two memories stored in this network? Then you would have two val¬ 
leys. This would mean that there was a set of patterns that would roll down into the first 
valley and a different set of locations that would roll down into the second valley. We 
call each of these sets the basin of attraction for a given valley. Just as water to the east of 
the Continental Divide in the Rocky Mountains in the United States flows east to the 
Atlantic, while water to the west of the Continental Divide flows west to the Pacific, 
starting points in one basin of attraction flow to one memory, while starting points in 
the other basin flow to the other (Figure C.3). 

What is really going on here is that the pattern of hills and valleys depends on the con¬ 
nections between cells. By changing those connections, we change the hills and valleys 
and change how our memory completes patterns. Storing a memory entails changing 


B The term “connectionism” is often limited to highly abstract models like the Hopfield model, 
which is obviously a very poor description of actual neurons. However, the original concept of con¬ 
nectionism was the general idea that it is the connections that matter and that information processing 
can occur by information flow across those connections . 14 

c One of the great ongoing debates in neuroscience is whether information is represented by the 
specific timing of spikes or by the average firing rate . 16 My suspicion is that both spiking and average 
firing rates contain information that is used by the brain. 

In some cases, it is clear that specific timing is critical. For example, in the bat echolocation system, 
spikes are timed to an accuracy of a microsecond (one millionth of a second), because the speed of 
sound means the bat has to recognize microsecond time differences . 17 In other cases, it is less clear. 
Depolarizations in cortical neurons, particularly in their inputs along their dendrites, do not reliably 
produce spikes, suggesting that spiking is more rate-based in the cortex . 18 

My take on this is that there is a continuum that depends on the neural structure involved. A useful 
thought experiment is that we can rephrase this question as “By how much time could you shift the 
spike randomly before the information was lost ?” 19 Obviously, even cortical spikes could not shift by 
seconds . 20 Similarly, the bat echolocation system needs spike accuracy to the microsecond . 21 But would 
the cortical information be lost if you shifted the cortical spikes by a microsecond? By a millisecond? 
By a centisecond? 
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ARCTIC OCEAN 



Figure C.3 Watersheds of North America as Basins of Attraction. The simplest 
example of a basin of attraction is that of a watershed. Water to the west of the Continental 
Divide ends in the Pacific Ocean, while water east of it ends in the Atlantic. Most of the 
Canadian water flows to the Arctic, but some flows to the St. Lawrence Seaway, and some 
flows to the Pacific. Water in the Great Basin flows into the Great Salt Lake and does not 
end up in an ocean. (Map modified from the U.S. Geological Survey.) 


those connections so that the memory is at the bottom of a valley. 23 Recalling a memory 
entails rolling down that surface to the bottom of the valley (Figure C.4). Changing 
connection weights changes the basins of attraction for each valley and changes what 
memories are recalled from what cues. It turns out that (as originally predicted by Hebb) 
the Hebbian synapse (or modifications of it that include both increases and decreases 24 ) 
changes the connection weights in the right ways to change the basins of attraction. In 
fact, many memories, with many valleys and many basins of attraction, can be stored in 
these networks. There’s good evidence now that several brain structures that store and 
recall memory work this way, including both the hippocampus and frontal cortex. 25 

Examples of the effects of these basins can be seen in how they affect categorization— 
for example, in speech, in the ability to recognize differences between sounds, or, in per¬ 
ception, in the ability to differentiate colors. 

Although human speech is capable of making many sounds reliably, each language 
selects a subset of those sounds to use to communicate information. 26 Thus, the Xhosian 
languages in southern Africa include different kinds of clicks (e.g., a slap of the tongue 
vs. a sounded swallowing), something not included in any Indo-European language. 
Some languages differentiate sounds that others don’t. For example, in Japanese, the 
sounds /l/ and /r/ (as in “glossary”) are both categorized as the same sound. Japanese 
words do include both /l/ and /r/ sounds, but native Japanese speakers don’t hear the 
difference. Every language has sounds it categorizes together. English, for example, 
doesn’t differentiate aspirated and unaspirated /1/ sounds—although English uses both 
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Figure C.4 Basins of Attraction. The basin of attraction is the set of points that, when 
allowed to change through a dynamic system, converge on the same place. If we think of 
the state of a neural system as a point in a high-dimensional space, then we can describe 
the flow of points in this space as a space of hills and valleys. Similar patterns will change to 
recall a stored pattern. 


aspirated and unaspirated consonants (for example, between the sound of the /t/ in the 
English words “ton” and “stun”), English speakers generally don’t hear the difference. 13 
In Hindi, the aspiration of certain consonants differentiates words (just as /l/ and /r/ 
differentiate “light” and “right”). We can explain this categorization ability in terms of 
basins of attraction 27 (Figure C.5). 

In actuality, the two sounds (/l/ and /r/) form a continuum in the shape of the 
mouth and the position of the tongue. /I/ is the tongue touching the front top of the pal¬ 
ate, while /r/ is the tongue against the side teeth. (The rolled /r/ of French or Spanish 
includes a glottal purring with the tongue against the side teeth.) In Japanese, all of the 
sounds in this continuum between /l/ and /r/ fall into the same basin of attraction; they 
activate the same cell assembly, and they are categorized as a single sound. In English, 
we draw a line between the extremes of this continuum, separating them. More /l/- 
like sounds fall into the /l/ basin, while more /r/-like sounds fall into the /r/ basin; 
they activate different cell assemblies, and they are categorized differently. This under¬ 
standing of categorization has led to new techniques for teaching Japanese speakers to 


D To prove to yourself that the /1/ is different in “ton” and “stun,” hold your hand a couple of inches 
in front of your mouth and say the two words. You'll feel your breath when saying the /1/ in “ton” but 
not in “stun.” 
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English speakers 
differentiate 
/L/ and /R/... 


butjapanese 
speakers don’t. 

Figure C.5 Phoneme-recognition Depends on Learned Basins of Attraction. 
The difference between being able to distinguish two categories depends on them being 
in different basins. Native English speakers have learned that the sounds /!/ and /r/ are 
in different basins, while native Japanese speakers have learned that they are in the same 
basin. This makes it very difficult for native Japanese speakers to recognize the differences 
between the /!/ and /r/ sounds. 



recognize the English differentiation between /l/ and /r/ by starting with the largest 
separation, thus overemphasizing the difference between them, and then explicitly 
training the categorization skills. 28 This enables Japanese speakers to create new basins 
of attraction, to learn to separate these categories, and to improve their ability to dif¬ 
ferentiate the sounds. 29 Similarly, parents talking to young children overemphasize the 
difference between individual vowels, which may help demonstrate the clear category 
differences to the children. 30 

Another classic example of learned categorization is color differentiation, which 
turns out to be helped by being able to name the colors. 31 Although there are univer¬ 
sal in some of the color labels between languages, likely due to the neurophysiology 
of color vision, every language differentiates colors a little differently. 32 (The nice 
thing about color is that one can use pure colors, which means that one can quan¬ 
titatively measure the difference between the two colors in terms of the wavelength 
of light.) English speakers are more easily able to distinguish colors at the boundary 
between blue and green than two colors that are both blue or that are both green, 
even when faced with color pairs that are separated by the same wavelength differ¬ 
ence. 33 This is a general property across languages. For example, while English has a 
single color term for blue (“blue”), Russian has two (chhhh “seenee,” dark blue, and 
rojiybon “goloboi,” light blue). This doesn’t mean that English speakers can’t separate 
light and dark blue; we often talk about sky blue and navy blue. These are small and 
subtle effects; certainly, we recognize the differences between the many blues in a 
big box of crayons. These effects should not be confused with the idea that some 
concepts do not exist in some languages, and that one cannot think about things one 
cannot name. 

For example, some people have argued that some languages do not represent time 
or number, and that those people live ever in the present. 34 This hypothesis is known as 
the “Sapir-Whorf” hypothesis (named after the linguists Edward Sapir and Benjamin 
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Whorf, not to be confused with the Klingon Worf) and is best known through the use 
of Newspeak in George Orwell’s 1984, where concepts are manipulated by changing 
the language itself. While it is clear that political discourse is changed by the framing of 
concepts and issues, it is still possible to think about things, even without terminology 
for them. 35 Similarly, the myth that the Inuit have twenty words for snow 36 should not 
be taken as so different from English. 37 English can also identify twenty different kinds 
of snow: sleet, ice, rain, snow, fluffy snow, heavy snow, etc. We certainly recognize the dif¬ 
ferences between these forms of snow (I know I do when I’m trying to shovel my drive¬ 
way!). The difference is that the Inuit tend to use more accurate words for snow because 
it is an important part of their lives. Similarly, while a casual observer may label a bird 
as “a bird,” a birdwatcher or a hunter would be more accurate, labeling it as a “hawk” or 
even a “red-tailed hawk.” 38 

Russian speakers have learned to categorize dark blue (chhhh, seenee) and light 
blue (rojiyhon, goloboi) rather than to categorize blue together. Native Russian speak¬ 
ers are faster to identify differences in blue colors that cross the chhhh (seenee)- 
rojiyhoH (goloboi) boundary than native English speakers. These abilities depend on 
language—if one is given a linguistic blocking task (like repeating random strings of 
numbers silently while doing the task E ), then the differences between color pairs cross¬ 
ing categories and color pairs within categories disappears. 39 

Presumably, the difference between native English and native Russian speak¬ 
ers is that the native Russian speakers have developed two basins of attraction to 
separate light and dark blue while the native English speakers have only one. What 
is particularly interesting about these categories is that they are not arbitrary; they 
relate to universal perceptual experiences and an interaction between our sensory 
perceptions and the world. 40 The cell assemblies in the color-recognition structures 
in our cortex are working from the specific color signals in the world (the blue of 
the sky, the blue of the ocean, the red of a sunset, the green of a leaf, the yellow of a 
sunflower) and the specific color signals that the photoreceptors in the retina of our 
eyes can detect/ 


E We’ve seen several cases throughout the book where a decision-making ability depends on 
using resources that are also used by linguistic or executive-function (conscious, requiring mental 
effort) tasks. If those resources are used by another task, then the ability disappears. The second 
task interferes with the first task. Good examples of linguistic and executive-function blocker tasks 
include being asked to remember a long random string of letters or numbers, saying the alphabet 
backwards, or counting back from 1000 by sevens. Try doing the last two tasks at the same time. 
It’s really hard! 

F Normal humans have three color-detecting photoreceptors called cones, each of which has a 
broad sensitivity to color (making this a distributed representation of color!). Subsequent calculations 
occurring in our retina and visual cortex compare the activity in contrasting pairs of color-detecting 
photoreceptors. This is why if you stare at a red picture long enough, when you look at a white screen, 
you’ll see a green afterimage. (Tire red cones have gotten tired and don’t fire as strongly as they should 
when you look away at the white screen.) Colorblind people are usually missing one or more of the 
cones, which is why people with red-green color blindness (missing the red photoreceptor) can dif¬ 
ferentiate blue and yellow but not red and green. The specifics of color perception would take up an 
entire book in itself, but for those interested, most neuroscience textbooks have a good description of 
the mechanisms of color perception. 41 
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The neurophysiology of cell assemblies 

Neurophysio logically, studying cell assemblies, content-addressable memory, and pat¬ 
tern completion requires the ability to record the activity of large numbers of neurons 
simultaneously, yet separately. It is not enough to see the broad activity of a network, as, 
for example, from EEG, LFP, or fMRI; one needs to see the actual pattern of neuronal 
firing. Additionally, it is not enough to see the activity of a single neuron because what 
matters is not whether a single neuron fires when a stimulus is presented or an action is 
taken; what is needed is to see how the activity of a single neuron reflects the activity of 
its neighbors. 

The first major steps toward the identification of cell assemblies in real nervous sys¬ 
tems were due to work by Bruce McNaughton and his colleagues in the 1990s, where 
they developed techniques capable of recording large ensembles of neurons simulta¬ 
neously. In a famous paper published in 1993, Matt Wilson and Bruce McNaughton 42 
recorded from over a hundred hippocampal place cells simultaneously. The hippocam¬ 
pus has been a particularly useful place to examine cell assemblies because hippocam¬ 
pal cells in the rat have a very identifiable representation—they encode the location of 
the animal in space. 43 This has enabled neuroscientists to use mathematics originally 
derived for spatial reasoning to examine hippocampal cell assemblies. 44 

More recently, it has been possible to examine patterns of cortical cells during action 
selection by examining the trajectory of the ensemble neural firing pattern through the 
high-dimensional space. As discussed above, we can imagine the population as a point 
in a very high-dimensional space, one dimension for each neuron. The activity of the 
cells at a moment in time can be described as a point in that space. Several labs studying 
action-taking have found that just before the animal takes the action, the firing pattern 
of the population of cells in motor cortex moves to a consistent point in that space and 
then takes a consistent trajectory to another point in that space. 45 Similar analyses have 
been done for ensembles of prefrontal neurons in rats—the firing pattern of the popu¬ 
lation of cells clusters into subspaces at important times, say when the animal changes 
between tasks. 46 

Just as Hebb predicted, the mental representation of decisions requires pattern com¬ 
pletion of cell assemblies and entails trajectories through cell-assembly space. 

Books and papers for further reading 

• George Lakoff (1990/1997). Women, Fire, and Dangerous Things. Chicago: University 
of Chicago Press. 

. Donald O. Hebb (1949/2002). The Organization of Behavior. New York: 
Wiley/Lawrence Erlbaum. 

• John Hertz, Anders Krogh, and Richard G. Palmer (1991). Introduction to the Theory 
of Neural Computation. Reading, MA: Addison-Wesley. 
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